sumo-agents 0.1.0 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,626 +1,112 @@
1
1
  # sumo-agents
2
2
 
3
- One repo you open and talk to. It works across every other project on the machine, remembers what you
4
- tell it so you never explain twice, learns each project as you go, and hands big work to cheaper
5
- jobs that run in its own loop against the Anthropic API. No harness: Sumo owns its prompt, its tools,
6
- its context and its cost.
3
+ A coding agent you talk to from one place. It works across every project on your machine, remembers what
4
+ you tell it, and hands big work to background jobs that are checked before they can say they are done.
7
5
 
8
- The repo holds only the process. Everything learned lives in `~/.sumo-agents/` on each machine and is
9
- never committed — a work laptop and a personal one learn separately.
6
+ - **One chat for everything.** Mention a project by name; Sumo finds it on disk and works there.
7
+ - **Memory that lasts.** Your preferences, project rules and where you left off are kept locally and
8
+ loaded into every new session. You never explain the same thing twice.
9
+ - **Background jobs.** Big, parallel or reading-heavy work goes to scout, worker and reviewer jobs. A
10
+ local model picks the model and effort for each one, so the dearest model is never the default.
11
+ - **Checked results.** A worker cannot report done until the project's own tests and linters agree.
12
+ - **Its own loop.** Sumo talks to the Anthropic API directly and owns its prompt, tools, context and cost.
10
13
 
11
- ## Set up a machine
14
+ Everything it learns lives in `~/.sumo-agents/` on each machine, never in a repository, so a work laptop
15
+ and a personal one learn separately.
12
16
 
13
- Needs Node 22.19+ on the 22.x line or Node 24.6+, llama.cpp (`brew install llama.cpp`) for the local model that
14
- routes jobs and writes memory, and, for the chat and jobs, either `ANTHROPIC_API_KEY` or a `CLAUDE_CODE_OAUTH_TOKEN`
15
- from `claude setup-token` in the environment.
17
+ ## Install
16
18
 
17
- ```sh
18
- npm install -g sumo-agents # puts `sumo` on your PATH
19
- sumo setup # creates ~/.sumo-agents, fetches the router model
20
- sumo doctor # every line should say ok
21
- sumo chat # talk
22
- ```
23
-
24
- `npm update -g sumo-agents` moves to the newest release. To work on Sumo itself, clone the repo instead:
25
- `git clone https://github.com/sumododda/sumo-agents.git && cd sumo-agents && npm ci && node bin/sumo.mjs setup`.
26
-
27
- Setup asks one question, once: where to download the router model from. Enter keeps the default,
28
- Hugging Face. Behind a corporate proxy, give the base URL of an Artifactory Hugging Face remote
29
- instead (`https://<host>/artifactory/api/huggingfaceml/<repo-key>`); the same
30
- `<repo>/resolve/main/<file>` path is fetched under it. The 2.5 GB file lands in
31
- `~/.sumo-agents/models/` and is skipped when it is already there.
32
-
33
- Sumo enables `NODE_USE_SYSTEM_CA=1` automatically, including for hooks and background processes.
34
- It trusts the OS certificate store alongside Node's bundled CAs, so enterprise CAs installed and
35
- trusted by IT work without a shell export. If IT supplies a separate PEM bundle, set
36
- `NODE_EXTRA_CA_CERTS` to its path before starting Sumo; that bundle is preserved. Restart existing
37
- Sumo processes after changing certificate trust.
38
-
39
- ```sh
40
- sumo setup --model-source https://<host>/artifactory/api/huggingfaceml/hf-remote # non-interactive
41
- sumo setup --no-model # skip the download
42
- sumo config model.source <url> # change it later
43
- sumo config chat.model opus # what `sumo chat` runs on; `auto` routes each turn
44
- sumo config chat.effort high # effort when using a fixed chat model
45
- sumo config chat.notify off # no bell when a long turn ends behind another window
46
- sumo config guides.auto off # no guide riding in unasked, no turn held for an unchecked edit
47
- sumo config jobs.advisor on # a job may ask the next model up for advice, mid-request (off)
48
- sumo config projects.home ~/code # where a new project is made (your home, until you say)
49
- sumo config scratch.days 30 # how long scratch keeps a chore nothing is written in (14)
50
- sumo models # the API models: on or off, and why
51
- sumo models discover # ask the API again which of them this credential can use
52
- sumo models disable fable # keep one out of the chat, the router and the passes
53
- ```
54
-
55
- Then just talk. Mention a project the way you normally would; the first time, the agent finds it on
56
- disk, confirms the path with you, and registers it — and a name of yours it had to look up becomes an alias,
57
- so it is looked up once. Work that belongs to no project is done in `~/.sumo-agents/scratch/`, a folder per
58
- chore: with no project named the tools start there, wherever you started the chat; a job is handed it as the
59
- project `scratch`; nothing is remembered or checked there, and a chore nothing is written in for 14 days is
60
- deleted — move one out and register it to keep it. In the chat: `/fix`, `/feature`, `/review`
61
- load the matching guide; `/dream` tidies the memory and says what changed; `/model` says the route
62
- and sets it — `/model opus high`, `/model haiku`,
63
- or `/model auto` to have the same local router that routes jobs pick a model and effort for each turn
64
- from what you typed (the menu offers the choices as you type; a switch costs one uncached turn, and a
65
- router that cannot answer refuses the turn); `/new` starts a fresh session, the memory block given to
66
- the model again, at the top of the window — the old one scrolled up, not wiped; `/resume` picks a saved session up where it stopped, its last five turns drawn again so you see where that was — every
67
- session is saved after each turn, whole but for its pictures and with secrets redacted, the menu lists
68
- them newest first with the heavy ones marked, a session open in another terminal is refused, and
69
- `sumo chat --resume [id]` opens one from the shell (the newest without an id); `/copy` puts the last reply on the clipboard as it was written — no indent, no line folded to the window — and `/copy code` its last code block; `/help` lists the commands and the keys; `/memory` opens the memory in your browser; `/mcp` lists the MCP servers — connected or not, their tools, what may run unasked; `/stop` stops a job at work in the background; `! <command>` runs a command yourself, where the agent would be refused; `/exit` (or `/quit`) ends it.
70
-
71
- The chat is a screen, not a scroll of lines: a box to type in (`\`+Enter, Shift-Enter or Option-Enter
72
- for a new line, Up for what you sent before — the last fifty, kept across sessions — `/` for the command menu;
73
- Ctrl-V pastes an image from the clipboard, on macOS, and an image file dropped on the box is the image —
74
- either stands in the box as `[Image #1]` and goes to the model with the message that names it; a paste too
75
- long for the box is drawn by the ten rows around the cursor, all of it still there; `@` and a few letters
76
- names a file of the project — the menu offers the ones they could mean, Enter or Tab puts the path in, Esc
77
- shuts the menu and keeps your words; Ctrl-G opens the message in `$VISUAL` or `$EDITOR`, and what you save
78
- there comes back to the box). What you said stands
79
- at full weight with a bar down its side. Each tool call shows what ran and what came back: a command by its
80
- first lines, and while it runs the last three lines it has printed and how long it has run; then the first
81
- two and the last three lines of its output, because a run says how it went at the end; an edit as a diff —
82
- the file from the project, the line it is on, `+1 −1`, the lines that changed numbered as the file numbers
83
- them with a line of what stayed around them. Ctrl-O is everything in full — the whole conversation in your
84
- pager (`$PAGER`, or `less`), every command whole with all its output, to scroll and to search; the screen
85
- behind it is left as it was. The reply's markdown is drawn as it reads —
86
- bold, italics, lists and task boxes, quotes, code blocks with their language and at full weight, tables as
87
- tables — and a working line while the model is busy, which says so when the model has been quiet for twenty
88
- seconds with nothing running. A delegated job stands under the call that started it while it works: how long,
89
- how many calls, what it runs now; it ends in one line — done or failed, what it cost — over the top of its report.
90
-
91
- A turn that made calls, or took ten seconds, ends on one line: how long, how many calls, the files it edited,
92
- what it cost. If it took ten seconds and the terminal says its window is not the one in front, it also rings
93
- — a bell and a notification (OSC 9), as does a question that waits for you; `sumo config chat.notify off`
94
- turns that off. The window's title says `sumo · <project>`, with a mark while it works or waits. Under the
95
- box: the model and effort, the size of the context, what the session has cost, and the project the tools run
96
- in with its branch.
97
-
98
- Esc stops the turn — first it closes an open command menu — and Ctrl-C first clears what is typed; Ctrl-C twice
99
- on an empty box leaves. A message sent while the model works is read with its next request, in the same turn
100
- (`next step:` above the box until it is); a command or a picture waits for the turn to end (`queued:`). Up on
101
- an empty box takes back what is waiting, and stopping the turn brings all of it back to the box instead of
102
- sending it. A resized window is redrawn to fit. The memory block goes to the model, not the screen; only a
103
- warning in it is shown. Opening the chat scrolls the window clear rather than wiping it, so what the terminal
104
- showed before is still in its scrollback; a resized window is drawn again from the top, and that does wipe it. While the
105
- conversation is short the box stands at the foot of the window; until you send the first message a lion stands
106
- faint in the room above it. The words on the working line are yours:
107
- `spinner.txt`, one a line, a random one to start each turn and the next every five seconds; so is the lion,
108
- `logo.txt`.
109
- Piped (`echo … | sumo chat`), it prints plain lines, the memory block first.
110
-
111
- ## Update a machine
112
-
113
- ```sh
114
- cd ~/sumo-agents && git pull
115
- npm ci # refresh dependencies from the lockfile
116
- sumo doctor # first line shows the commit now running — compare it with GitHub
117
- ```
118
-
119
- `sumo` runs straight from the repo, so new code is live the moment it is pulled, and the
120
- database upgrades itself the next time it is opened. Your memory is not touched — it lives in
121
- `~/.sumo-agents/`, outside the repo. Run `sumo setup` again only if `sumo doctor` tells you to (you moved
122
- the repo, or changed Node versions). Start a new `sumo chat` afterwards, so it loads the updated code
123
- and `AGENTS.md`.
124
-
125
- ## What happens while you talk
126
-
127
- Every session runs in Sumo's own loop: one request per turn to the Anthropic Messages API, three tools
128
- (a shell and a file editor, both Anthropic-defined so no schema is sent, and `delegate`, which hands work
129
- to a sub-agent) plus the tools of any MCP servers you configured, deferred behind a search tool so
130
- none of their schemas sits in the context until it is needed (see *MCP servers*), a frozen system prompt that
131
- caches across turns, and old tool results cleared server-side once the context passes 60k tokens.
132
- History is never rewritten on the client. A request the API or the connection lets down — an overload, a
133
- rate limit, a stream that breaks half-way — is sent again, up to six times with a wait that doubles from a
134
- second to thirty (or what the API asked for), each try said on the screen and written to the ledger; a
135
- request the API refuses as wrong is not. A turn the API pauses of its own accord goes on. The same call made
136
- with the same answer three times in a row is said to the model; five times, and the turn is stopped. The turn
137
- before the last of a run carries a warning, so a job that hits the turn limit still closes. Every model but
138
- Haiku is asked to say what it is thinking in summary, and the end of it — its last three rows, folded to the
139
- window — shows under the working line.
140
-
141
- | When | What runs | Cost to the model you chat with |
142
- |---|---|---|
143
- | Session starts | The first user turn is the core block: your global preferences, workflows, projects, where you left off, open jobs, things to confirm. Default cap 800 tokens (`prime.budget`); overflow is reachable by search. | ≤ 800 tokens by default, once |
144
- | You send a message | It is stored word for word (secrets redacted). First mention of a project adds its card — path, stack, commands, your rules for it, gotchas — as an operator message after the cached prefix. | 0, or ≤ 200 once per project |
145
- | A shell command matches a destructive or secret-reading guard (`rm -rf ~` — by any path, escaped, behind `sudo`, `env`, `nice`, `timeout` or `xargs`, flags before or after the target — `git reset --hard`, `git clean -f`, `DROP TABLE`, `cat .env` …), or the editor tries to access a secret file | Refused in plain code before it runs, and the refusal says why. No memory is consulted. The way through is you: `! <command>` runs it yourself. A force-push is not on the list — it is an accepted way of cleaning up history here. | 0 |
146
- | A command prints more than the model is shown (16k characters) | The two ends are shown, and the first and last 64 KB are written — without secrets — to the job's folder (or `~/.sumo-agents/logs/output/` for the chat), named in the cut, so the model reads a range of the file instead of running the command again. Kept a week. | 0 |
147
- | An edit's `old_str` is not in the file exactly as written | It is matched again with the file's line endings, then line by line without trailing spaces, then without its indentation too — the replacement set in at the file's depth — and the result says how it matched. A looser match is still refused unless it is the only one. | 0 |
148
- | The editor reaches for a path outside the project | Refused: the editor is jailed to the project (symlinks followed). Shell commands start in the project directory but can access other paths, and run with your environment — every token you exported, so a probe or a smoke test can use it — minus Sumo's own Anthropic credential. A command whose only job is to print the whole environment (`env`, `printenv`, bare `export` or `set`) is refused; a `.env` is loaded with `set -a; source .env; set +a`, never printed. | 0 |
149
- | The agent is about to run a shell command a taught workflow gates (`gh pr create`, for a workflow taught with `--gate 'gh(-axi)? pr create'`) | The command is held back once and the agent is handed the workflow's steps. It follows them, then runs the command. The same steps ride in with your message when you ask for the thing yourself. | 0 until it fires |
150
- | Your message asks for a fix, a feature or a review ("the export is broken", "add a --json flag", "review j12") | The matching guide rides in as an operator message, once per session — judged by the same word stems as a taught workflow, with no model asked. A taught workflow that matches the same message goes in instead. `/fix`, `/feature` and `/review` name one outright. The screen shows `guide: fix`. | 0, or ≤ 450 once per guide |
151
- | The agent ends a turn after editing a file, and none of the project's declared checks (`npm test`, `make check` …) ran since | It is told the fact — which files, which check — and continued once, the way a question memory can answer is. A command that is not the check (`git status`) does not count; a project that declares no check is cleared by any command. The screen shows `held: 2 files edited, npm test not run`. | ≤ 60 tokens and one request, on such a turn |
152
- | The agent ends a turn on a question | Memory is searched with the question's own words, in plain code. If something close is there, the agent is handed it once and carries on instead of waiting for you; if nothing is, the question reaches you untouched. | 0 unless memory answers |
153
- | A turn ends | The **scribe** is scheduled when a pending turn contains a standing instruction, three turns are waiting, or its last run was at least five minutes ago (or it has never run). Session start and end also sweep up pending turns. A detached call to the local model proposes memories; code checks each against what you actually typed. | 0 |
154
- | 3 finished sessions pile up | The **dream** pass reads them side by side: contradictions, repeated corrections, routines worth naming. | 0 |
155
- | Every model response | One ledger row: tokens in, cached, written, out, cost, the job or session it served. `sumo scribe stats` sums them. | 0 |
156
-
157
- What you state is saved silently. What the model could not tie to your exact words is held and put
158
- to you as a one-line question at the next session start. Nothing from web pages, files or tool output
159
- can become a memory.
160
-
161
- Corrections that differ by one word or a case-sensitive name are kept as distinct statements;
162
- similar memories can be flagged for you to settle. One message can also state separate rules for
163
- several projects. Background passes serialize their runs and mark only the turns they actually
164
- read as processed, so a message arriving during a pass remains pending. Dream waits for the writer
165
- to finish filing a session before consolidating it.
166
-
167
- In `sumo memory`, unsaved edits survive filtering, scope changes and a refused save within the page.
168
- A successful save or Cancel clears that draft; reloading or closing the page loses it.
169
-
170
- ## See what it knows
171
-
172
- ```sh
173
- sumo memory # the same, as a page in your browser: say yes or no to guesses, edit, forget
174
- sumo export # everything, readable, grouped by project
175
- sumo prime # exactly what a new session is shown
176
- sumo project show <name> # a project's card
177
- sumo search "<words>" [--project <name>]
178
- sumo search "<words>" --turns # what you literally typed
179
- sumo history <id> # what a memory replaced, and what replaced it
180
- sumo gate <workflow-id> '<regex>' # which shell commands a workflow must come before (off to remove)
181
- sumo scribe show # exactly what the cheap model would be sent right now
182
- sumo scribe stats # what the background passes have cost
183
- sumo forget <id> # stop it being true (history kept); --purge erases it for good
184
- ```
185
-
186
- `sumo help` lists the rest. Ids look like `m12` (memories) and `j3` (jobs).
187
-
188
- ## Delegation
189
-
190
- The main agent does small things itself. Big, parallel or reading-heavy work it hands off with its
191
- `delegate` tool: a written brief becomes a job, and the job runs in the chat's own process on the route the
192
- router chose. On the chat's screen it runs in the background: the call answers at once, the turn goes on or
193
- ends, and you keep working — ask something else, start another job. The job stands in a row under the box
194
- while it works (`⏺ j43 scout — Map the parser · 2m 10s · 14 tool calls · working`) — what it runs and thinks is
195
- not said there. Over the jobs stands a row for the chat itself (`⏺ main · working`). Down from the box steps onto a
196
- job and Enter opens it: the end of its record above the box — each call, the top of what came back, what it
197
- said — as it grows; `@ message` then talks to that job, Ctrl-O shows all of the record, Esc closes it — or Down and
198
- Enter on `main` comes back to the chat. When
199
- it reports, how it went is drawn where the conversation has got to, and the model reads the report as an
200
- automated event, never as your words: with its next request if it is at work, otherwise in a turn the
201
- screen starts for it — so the main agent picks the work back up by itself. Esc stops the chat's turn, not
202
- the job; `/stop` (or `/stop j43`)
203
- stops a job, which says so with your next message and starts no turn; a question a job asks is answered or
204
- refused (Esc) without stopping it; `/new` and `/resume` wait until no job is at work; leaving the chat stops
205
- the jobs still running, to be picked up again later. Piped, and in `sumo job run`, nothing changes: the call
206
- waits and the report is its result. Several `delegate` calls in one reply run at the same time. The job gets the brief as its only user turn — with the project's card and your global
207
- rules in it — a short frozen system prompt that carries the Coding rules from `AGENTS.md`, and tools by role:
208
-
209
- | Role | Model and effort | Tools | For |
210
- |---|---|---|---|
211
- | `scout` | chosen per job by the router | shell + web search and fetch + job tools | finding, tracing, auditing, summarizing — anything reading-heavy |
212
- | `worker` | chosen per job by the router | shell + editor + `baseline`/`verify` + job tools | building and fixing |
213
- | `reviewer` | chosen per job by the router | shell + job tools | judging a change it did not write |
214
-
215
- The job tools are `note`, `ask`, `search_memory` and `finish`. Each runs the `sumo job` command a person
216
- would, in a process of its own, so a worker's checks never stall the chat or the jobs beside it. A job that
217
- stops without calling `finish` is told once to close itself. The Anthropic credential never enters a shell:
218
- `sumo job run` and `sumo job new` typed into the chat's shell are refused with a pointer to `delegate`.
219
- While jobs run the chat shows them: which job, on what model, each thing it runs (one line each; Ctrl-O for
220
- the output) and what it says. `@ …` is said to the one job running, `@j<id> …` to a named one — read with
221
- its next request; a job running elsewhere is reached on disk, as with `sumo job tell <id>`. Briefs, notes,
222
- answers, reports and check results live in `~/.sumo-agents/jobs/<id>/`, so a job started today can be
223
- picked up tomorrow: `delegate` with only its `job` id, or `sumo job run <id>` in a terminal. One worker per
224
- project at a time — they share a working tree.
225
-
226
- ## The web
227
-
228
- The chat and a `scout` can search the web and read a page: `web_search` and `web_fetch`, the API's own tools. They
229
- run on Anthropic's side, inside the request — no key, no dependency, nothing run on this machine. A search shows as
230
- `Search(…)` with the pages found under it, a page read as `Fetch(…)` with its size, and the pages a reply cites are
231
- listed under it. Workers and reviewers do not have them: a page is somebody else's words, and it is kept out of
232
- the conversations that change code or judge it. Nothing here enforces that a page's words stay information: the
233
- chat that reads one has a shell with your environment, and only the model's judgment stands between them.
234
-
235
- A search costs $0.01 on top of its tokens and is counted in the ledger; a fetch costs its tokens only. One search
236
- and one page read is some 30,000 tokens of context, so a lookup across many pages belongs to a scout. A page is cut
237
- at about 20,000 tokens; a PDF comes whole. There is no cap on how many searches a reply makes. Both tools are called
238
- directly: the newer way, where the model filters results in code first, is refused on a Claude Code token.
239
-
240
- ## MCP servers
241
-
242
- Sumo talks to the MCP servers you configure — Azure DevOps, Slack, whatever you use — and nothing is set up
243
- by default. They live in `~/.sumo-agents/mcp.json`, in the shape Claude Code uses, so a config you already
244
- have pastes in; `sumo mcp add` writes it for you:
245
-
246
- ```sh
247
- sumo mcp add azure-devops -- npx -y @azure-devops/mcp <your-org> # run as a process (stdio); it signs in through `az login`
248
- sumo mcp add slack --env 'SLACK_TOKEN=${SLACK_TOKEN}' -- npx -y <the slack server> # a token the server needs, named — never written
249
- sumo mcp add remote --url https://host/mcp --header 'Authorization=Bearer ${TOKEN}' # reached over streamable HTTP
250
- sumo mcp # every server: where it is, its tools or why it failed, what may run unasked (/mcp in the chat)
251
- sumo mcp tools azure-devops # one server's tools
252
- sumo mcp allow azure-devops wit_get_work_item # let it run without asking — `all` for the whole server; `revoke` undoes it
253
- ```
254
-
255
- `${NAME}` in a value is filled from the shell when the server starts, so the file never holds a token, and a
256
- name nothing fills refuses that server with the reason. A stdio server gets your environment without Sumo's
257
- own credential, plus the entries you name — so a server that reads its token from a variable you exported just
258
- works, and `env` is for one it wants under another name — and the node that runs Sumo first on its PATH, so
259
- `npx` is found. Give commands and paths in full: a server starts where Sumo was.
260
-
261
- The model sees the tools of every connected server, deferred behind Anthropic's tool search: none of their
262
- schemas enters the context until the model searches for one by name or description, so forty Azure DevOps
263
- tools cost nothing per turn until one is used, and the cached prefix is untouched. The system prompt names
264
- the servers and how many tools each has. The chat and every job get them alike — a job started from a
265
- terminal too.
266
-
267
- Every call is held to an allow list. A tool not on it stops and asks on the chat's screen — `y` runs it
268
- once, `a` always, `n` refuses it, Esc stops the turn — and `a` is written to the file. A job delegated from
269
- the chat asks on the same screen, with its id on the question. `sumo job run` in a terminal, or the chat
270
- piped, has nobody to ask, so the call is refused with the `sumo mcp allow` line that would allow it. A
271
- refusal is final for that call: the model is told not to try it again.
272
-
273
- What comes back passes through the same cap and redaction as any tool output; a picture or a file is
274
- described, not sent. A server that dies is started again on the next call; one that fails to connect is
275
- said on screen and simply absent for that session. Servers connect once per `sumo chat` process; a change
276
- to the file is picked up by the next one.
277
-
278
- Not in this version: the old HTTP+SSE transport, signing in to a remote server (give it a token in a
279
- header), pictures to the model, MCP resources and prompts.
280
-
281
- ## Which model and effort a job gets
282
-
283
- The model you chat with never decides this. Making the job does, in plain code; `sumo job new` prints the result:
284
-
285
- ```
286
- created j27 [worker·sumo-agents·running] Session hygiene: context gauge …
287
- route: opus/xhigh — router: a new feature across hooks and sessions with tests
288
- run it: sumo job run 27 (append & to carry on without waiting for it)
289
- ```
290
-
291
- The route is the router's answer, and nothing else's. On every `sumo job new` and every `sumo job retry`, a
292
- 4B open model (Qwen3-4B, through llama.cpp) reads the role, project, task text and the project's history,
293
- describes the job — its kind, how much code it touches, whether it is risky — and then answers with a
294
- model, an effort and one sentence. It runs locally, costs nothing, and takes about a second once the
295
- model is loaded. A scout's job is put to it as reading, not as changing code: read as a change, an audit
296
- is "novel" and so is a summary of an unfamiliar project, and both went to the dearest model; read as
297
- reading, finding and describing are routine and only judging — an audit, a review, a bug hunt — is hard. There are no flags to override it, no project rules, no retry ladder and no floors.
298
-
299
- The same model, through the same server, writes memory; see *The local model* below for what it is, how it
300
- is run and how it was measured.
301
-
302
- The history it reads is only the evidence against a route: a model/effort that failed here, or that a
303
- review found Important issues in. A clean record is left out — it says nothing about what a cheaper
304
- route would have done, and a router shown one keeps choosing it.
305
-
306
- If the router is missing, fails, or answers something no job can run on (a model outside the list, or
307
- no effort for a model that takes one), the command exits with the error and **no job is created**. The
308
- answer schema pairs Haiku with no effort and every other model with one. The answer is taken as it is, for
309
- every role; the router is offered only the models that are on, so `sumo models disable haiku` keeps every job off it.
310
- `sumo doctor` fails while the router is missing.
311
-
312
- Effort is real, not advisory: the route's effort goes on the request as `output_config.effort`, its model as the
313
- model id, and the run's ledger rows say what ran — so what was chosen is what runs.
314
-
315
- Models available to jobs: `haiku`, `sonnet`, `opus`, `fable`. Effort levels: `low`, `medium`,
316
- `high`, `xhigh`, `max`; Haiku takes none.
317
-
318
- Each of the four has a switch on this machine, and nothing assumes the API has all of them. `sumo setup`
319
- asks the API once which of them this credential can use (one `GET /v1/models/<id>` per model) and turns
320
- off the ones it does not have; `sumo models` lists them with the API id and the reason for each switch;
321
- `sumo models discover` asks again and sets the switches from the answer, except that a model turned off by
322
- hand stays off; `sumo models enable|disable <name>` sets one by hand. A check that fails — no credential, an error, or an answer with none of them —
323
- changes nothing and says why. A model that is off is outside the router's grammar and prompt, refused by
324
- `/model` and by `sumo config chat.model` / `scribe.model` / `dream.model`, never retried on by the passes
325
- (the retry goes to the cheapest model that is on, or nowhere), and a job routed to it before it was
326
- turned off is not run. `sumo doctor` says which are on, warns while nothing was checked, and fails when
327
- none is on or a configured one is off.
328
-
329
- **The label that grades a route is a review, not a green check.** A worker's DONE only proves the
330
- checks passed; every Sonnet worker here passed its checks and the Opus reviews that followed found 1 to 7
331
- Important issues each. So when a reviewer finishes, its Important count is stored on the job it judged,
332
- and that is what the router's history and `sumo job stats` report:
333
-
334
- ```sh
335
- sumo job stats --project <slug> # per role and model/effort: jobs, done, failed, reviewed, avg important — and what a job took: requests, time, money
336
- sumo job retry <id> # a failed job, or one reviewed with 3+ Important: same brief,
337
- # previous notes and report attached, routed afresh
338
- ```
339
-
340
- The router itself is graded by a set of labelled tasks, each with the cheapest and the dearest route that
341
- would be right for it. Run it after changing the router's prompt, its history or its model; it asks the
342
- real router in a throwaway home and prints how many tasks were routed too high and too low:
19
+ You need Node 22.19+ or 24.6+, [llama.cpp](https://github.com/ggml-org/llama.cpp) (`brew install llama.cpp`)
20
+ for the local model, and `ANTHROPIC_API_KEY` or a `CLAUDE_CODE_OAUTH_TOKEN` (from `claude setup-token`) in
21
+ your environment.
343
22
 
344
23
  ```sh
345
- node --disable-warning=ExperimentalWarning test/router-probes.mjs
346
- ```
347
-
348
- Six of its eighteen tasks are scout jobs, labelled from this machine's own ledger: the haiku scouts took 39 to 77
349
- requests and $0.15 to $0.50 a job, the one on sonnet/low did a like audit in 6 requests for $0.07 — so for reading
350
- that is more than one lookup, haiku is not the cheap end. Measured 2026-10-10: 15, 15 and 16 of 18 right over the
351
- three histories; before scouts had a prompt of their own, 14, 14 and 15, with a summary of a project routed to fable.
352
- `sumo job stats` shows the same thing as it accrues — what a job on each route took, per role.
353
-
354
- A retry is routed like any job, except that the router is also told what the previous attempt ran on
355
- and how it ended.
356
-
357
- ## How coding work gets done
358
-
359
- Three kinds of work have a written way of doing them. Each is a short numbered list in `guides/`, and
360
- each rides in by itself when your message asks for that kind of work:
361
-
362
- | You type (or say) | Guide | The gist |
24
+ npm install -g sumo-agents
25
+ sumo setup # creates ~/.sumo-agents and downloads the 2.5 GB router model
26
+ sumo doctor # every line should say ok
27
+ sumo chat
28
+ ```
29
+
30
+ `npm update -g sumo-agents` moves to the newest release. Setup flags, corporate proxies and certificates
31
+ are covered in [Install](https://github.com/sumododda/sumo-agents/wiki/Install).
32
+
33
+ ## Your first session
34
+
35
+ - Just talk. Name the project you mean; the first time, Sumo confirms its path and registers it.
36
+ - Say how you like things done. "Always run the linter before you commit" is remembered in your words and
37
+ loaded next time.
38
+ - `/fix`, `/feature` and `/review` load a short written procedure for that kind of work. They also load on
39
+ their own when your message asks for one.
40
+ - Big work is delegated without being asked. A row for the job appears under the box, you keep talking,
41
+ and its report comes back into the conversation.
42
+ - `/new` starts a fresh session, and a new task deserves one. `/resume` picks an old one up. `/help` lists
43
+ the rest.
44
+
45
+ ## Good to know
46
+
47
+ - Destructive commands (`rm -rf ~`, `git reset --hard`, dropping a table) and commands that print secrets
48
+ are refused in code before they run. `! <command>` runs one yourself.
49
+ - On macOS the agent's shell runs in a sandbox: it writes only in the project, the temporary folders, Sumo's
50
+ home and the tool caches, never in git's hooks or config. A write it blocks is put to you once to run
51
+ outside it. `sumo config shell.network off` also keeps it off the network.
52
+ - The agent's shell gets your whole environment except the Anthropic credential. It never waits on a prompt,
53
+ stops a command silent for 2 minutes (`timeout 600 <cmd>` asks for longer), and runs a command ending in
54
+ `&` in the background with its own log.
55
+ - A worker's first DONE is answered with its task beside its change to check; the second runs your checks.
56
+ A failed first attempt is retried once on the next model up, and risky work is reviewed, without asking.
57
+ A second worker on a git project works in a worktree of its own.
58
+ - A job's run stops at $10 or 60 minutes (`jobs.budget`, `jobs.minutes`). `sumo cost` shows where the money went.
59
+ - `/undo` puts back the files the last chat turn changed with the editor.
60
+ - Only your words become memory. Web pages, files and tool output are never filed as yours.
61
+ - The local model, Qwen3-4B through llama.cpp, routes jobs and writes memory for $0 and sleeps when idle.
62
+
63
+ ## Configuration
64
+
65
+ `sumo config <key> <value>` sets one; `sumo config` lists them. The ones most people touch:
66
+
67
+ | Key | Default | What it does |
363
68
  |---|---|---|
364
- | `/fix <what is wrong>` | `guides/fix.md` | What changed recently → make it fail on demand → find *where* it breaks before saying *why* → one cause at a time, each with evidence → fix where it starts → the same reproduction passes |
365
- | `/feature <what to build>` | `guides/feature.md` | Size it (experiment · change to existing code · new subsystem) → agree what done means → tests first, seen failing → build without touching them → name a wrong implementation that would still pass, and kill it |
366
- | `/review <what to judge>` | `guides/review.md` | A reviewer that did not write the code, reading only the change: first "does it do what was asked" (missing · extra · misunderstood), then "can it be trusted"; every finding has a file:line and a concrete way it fails |
367
-
368
- You do not have to type the command. Each guide opens with a `when:` line — the phrases that mean you are
369
- asking for its kind of work (`fix, bug, broken, failing, crash, …`) — and code matches your message against
370
- them by word stems, the way a taught workflow is matched: no model, no delay, nothing in the context until
371
- one fires. One guide per message, review before fix before feature ("review my fix" is a review; "fix the
372
- broken add button" is a fix), once per session; a taught workflow that matches the same message wins. In
373
- the chat, `review` goes in as "delegate it to a reviewer" plus how to receive the findings, since code
374
- written in a session is never judged in that session. `test/guides.test.mjs` holds the probe set — fifty
375
- messages in the shape of real ones, each with the guide it should bring — and names the misses it accepts.
376
- `sumo config guides.auto off` turns the matching off; the commands still work.
377
-
378
- **`--guide`** is how the same steps reach a job. `sumo job new --guide fix` pastes `guides/fix.md`
379
- into the worker's brief, so a delegated fix is done the same way as one done in the conversation. Unsaid,
380
- a worker gets the guide its title and goal ask for, if any (`--guide none` for none); a reviewer job always
381
- carries `guides/review.md`; a scout gets none.
382
-
383
- ## How a worker's DONE is checked
384
-
385
- Whoever did the work does not grade it. A worker's `DONE` rests on what the project's own checks say,
386
- run by `sumo` — not on what the report says.
387
-
388
- ```
389
- sumo job new records where the work starts: the tree exactly as it is, your uncommitted
390
- edits included, so they are never counted as the job's
391
- sumo job baseline <id> the worker's first step — runs the project's checks before any edit and records
392
- what already fails
393
- … the worker works …
394
- sumo job verify <id> runs the same checks and gives the verdict, without closing anything
395
- sumo job finish <id> --status DONE
396
- uses that verdict if the files have not moved since, otherwise runs the checks
397
- again — and refuses DONE while anything is blocking
398
- ```
399
-
400
- Which commands are "the project's checks": its `check` script or Makefile target when it has one (that
401
- is the project's own gate), otherwise its `test`, `lint` and `typecheck` commands — the same ones shown
402
- on the project card. Each gets nine minutes (`SUMO_AGENTS_CHECK_TIMEOUT_MS`); the end of its output is
403
- kept in the job's folder.
404
-
405
- | The verdict says | Meaning | Effect |
406
- |---|---|---|
407
- | a check `FAILED` that passed at the baseline | the job broke it | **blocks DONE** |
408
- | a check `FAILED` and no baseline was taken | nothing shows it was already broken, so it counts as the job's | **blocks DONE** |
409
- | a test that already existed was edited, renamed or deleted | tests judge the change; they are not part of it | **blocks DONE** — unless the job was created with `--tests-may-change` |
410
- | a check declared at the start was removed, or its definition was changed | the original checks must still judge the work | **blocks DONE** — `--tests-may-change` allows definition changes, not removed commands |
411
- | files changed while the checks ran | the verdict does not cover the resulting tree | **blocks DONE** — verify again after the files settle |
412
- | a check `was already failing` | broken before the job began, and still is | allowed; both outputs are kept to compare |
413
- | a key, token or private key was added (`ghp_…`, `sk-…`, `AKIA…`, a PEM block, a JWT), or a secret file (`.env`, `*.pem`, `id_rsa`) is in the change | a credential is about to be committed | **blocks DONE** |
414
- | `look at:` added lines with `eslint-disable`, `@ts-ignore`, `# noqa`, `.skip(`, `t.Skip(` … | a checker was told to look away | allowed, and printed beside the STATUS line for a person to judge |
415
- | `look at:` added lines with `password=`, `token=`, or a long random string | a credential, or a fixture that looks like one | allowed, and printed for a person to judge |
416
- | `note:` not a git repository / no commands | it could not be known | said, never guessed |
417
-
418
- A refused worker always has an honest way out, and the refusal names it: fix it, finish as `FAILED`, or
419
- `sumo job ask`. When the checks cannot run on this machine, `--accept "<why>"` takes the work anyway and
420
- marks it `DONE (UNVERIFIED …)` in the STATUS line and in the report. So a plain `STATUS: DONE` from a
421
- worker means the checks agreed.
422
-
423
- A baseline asked for after the first edit is refused: it would call the job's own breakage "already
424
- there". New test files are always welcome — only tests that were there before the job are protected.
425
-
426
- The worker's report has two sections for things that would otherwise go unsaid: **Concerns** (finished,
427
- but not sure of) and **Decisions** (each call made on your behalf: what, why, what it costs if wrong).
428
- A running worker settles small questions itself and lists them; it stops to ask only for something
429
- destructive, security-sensitive, outside the project, or too unclear to do without guessing.
430
-
431
- ## Reviews
69
+ | `chat.model` | `opus` | What the chat runs on. `auto` lets the local router pick per turn. |
70
+ | `chat.effort` | `medium` | Effort when the chat model is fixed. |
71
+ | `chat.budget` | `5` | Dollars a chat session costs before the screen says so; `0` never. |
72
+ | `projects.home` | `~` | Where new projects are created. |
73
+ | `jobs.budget` | `10` | Dollars one job may spend before it is stopped; `0` for no cap. |
74
+ | `jobs.minutes` | `60` | Minutes one job may run before it is stopped; `0` for no limit. |
75
+ | `jobs.escalate` | `on` | A worker's failed first attempt is retried once on the next model up. |
76
+ | `jobs.review` | `risky` | Which verified work gets a reviewer without asking: `risky`, `all` or `off`. |
77
+ | `jobs.advisor` | `off` | `on` lets a job ask the next model up for advice. |
78
+ | `shell.sandbox` | `on` | The agent's shell in the macOS sandbox. Read when a chat or job starts. |
79
+ | `shell.network` | `on` | `off` keeps the sandboxed shell to this machine. |
80
+
81
+ The full list, the API model switches and the environment variables are in
82
+ [Install](https://github.com/sumododda/sumo-agents/wiki/Install#configuration).
83
+
84
+ ## Learn more
85
+
86
+ The [wiki](https://github.com/sumododda/sumo-agents/wiki) has the detail:
87
+
88
+ - [The chat](https://github.com/sumododda/sumo-agents/wiki/The-chat): commands, keys, the screen, sessions
89
+ - [Projects and memory](https://github.com/sumododda/sumo-agents/wiki/Projects-and-memory): cards, scratch, how memory is written and managed
90
+ - [Jobs](https://github.com/sumododda/sumo-agents/wiki/Jobs): delegation, routing, coding guides, verifying DONE, reviews
91
+ - [Web search and MCP](https://github.com/sumododda/sumo-agents/wiki/Web-search-and-MCP)
92
+ - [Safety](https://github.com/sumododda/sumo-agents/wiki/Safety): what is enforced by code and what is only asked
93
+ - [How it works](https://github.com/sumododda/sumo-agents/wiki/How-it-works): a turn, the local model, costs
94
+
95
+ ## Development
432
96
 
433
97
  ```sh
434
- sumo job new --project <slug> --agent reviewer --reviews <job id> --title "review j12" # that job's change
435
- sumo job new --project <slug> --agent reviewer --title "review the tree" # whatever is uncommitted
98
+ git clone https://github.com/sumododda/sumo-agents.git ~/sumo-agents && cd ~/sumo-agents
99
+ npm ci
100
+ node bin/sumo.mjs setup # points `sumo` at this clone instead of the package
101
+ npm test # offline suite, no real model calls
102
+ npm run build:bin # optional: a standalone binary in dist/sumo-agents/ (needs Bun); move the folder whole
436
103
  ```
437
104
 
438
- What was asked for goes on stdin — a review is against that, not against taste. The reviewer is handed
439
- the whole change as one file (`changes.diff` in its job folder; `sumo job changes <id>` writes the same
440
- file for any worker job), the original brief, and the author's report marked as *claims, not facts*.
441
- It reports: asked vs built · findings, worst first · minor (listed, never a reason to reopen the work) ·
442
- what the change alone could not show.
443
-
444
- The reviewer runs in a fresh context, with its model and effort chosen by the local router for that
445
- review. There is no guarantee that its model is stronger than the worker's. Findings are still
446
- claims — `guides/review.md` ends with how to receive them.
447
-
448
- ## When to start a new session
449
-
450
- Model quality falls with the absolute size of the context, not with the share of the window that is
451
- full: degradation is measurable from tens of thousands of tokens, and a coding-agent study passed 8 of
452
- 10 runs clean and 3 of 10 with about 75k tokens of extra context, relevant or not. A new task gets a
453
- new session. So:
454
-
455
- - **A gauge, not a guess.** The exact context size comes back with every response and shows in the
456
- chat's status line (or the plain-text prompt, `sumo 92k>`). At 80k tokens the agent is told once —
457
- finish the piece in hand, then ask for `/new`. At 150k it is told to stop and note where it is.
458
- - **A boundary nudge.** `sumo job finish` and `sumo job abandon` end with a `/new` line when the session
459
- is already heavy, because a task boundary is the cheapest place to start over.
460
- - **Nothing is lost.** The end of a turn records where you left off, and the next session's first
461
- block brings it back. Open jobs resume from their briefs. The session itself is still there too:
462
- `/resume` brings it back whole, for the price of one uncached turn, for thirty days.
463
- - **Old tool output goes first.** Past 60k tokens the API clears the oldest tool results from what
464
- the model reads, keeping the last five; the conversation itself stays intact.
465
- - **A job that outgrows the context is summarized, not stopped.** At 150k tokens, at the end of a tool
466
- round, the job's conversation so far is summarized server-side — told what to keep: the task, the
467
- decisions, what was tried, what is done, the next step, the exact names and errors — and the job goes
468
- on from the summary, with `sumo job brief <id>` named for whatever it left out. Not again within three
469
- turns; a summary that cannot be had costs nothing but its request. The chat keeps its nudges: a new task
470
- gets a new session.
471
- - **A resumed session carries no thinking.** A thinking block is bound to the exact history it was made
472
- in, and a saved session had its secrets redacted and its pictures taken out — so `/resume` sends the
473
- words and the calls, not the thinking, and the first reply reasons afresh.
474
-
475
- ## What a project card tells you about hygiene
476
-
477
- `sumo project add` scans what the project declares about itself. Besides stack and commands, the card
478
- says what is absent — `not set up here: a lint command, a CI workflow` — so that "nothing failed" is
479
- never mistaken for "it was checked". A `check` script or target counts as the project's gate and stands
480
- in for test and lint. `sumo project rescan <slug>` after setting one up.
481
-
482
- ## What is enforced, and what is only asked
483
-
484
- | Enforced by code | Asked in a prompt |
485
- |---|---|
486
- | a worker's DONE against the project's checks and a baseline | the steps in `guides/fix.md`, `feature.md`, `review.md` |
487
- | existing tests untouched by a worker | the rules in `AGENTS.md` for work done in the conversation itself |
488
- | reviewers and scouts receive no editor tool | reviewers and scouts use their shell only for reading |
489
- | | a worker running `sumo job baseline` first — though skipping it only hurts the worker |
490
- | a taught workflow's steps before a gated shell command | the nudge to start a new session |
491
- | the end of a turn that edited a file and ran no declared check — held once, the check named | the guide that rode in being followed |
492
- | no job, no run | |
493
- | matched destructive and secret-reading shell commands are refused; editor paths stay inside the project and exclude secret files | shell commands stay inside the project |
494
- | Sumo's own credential is withheld from every command; a command that prints the whole environment is refused; recognized secrets in output are redacted | |
495
- | a key or token in a worker's change blocks DONE | |
496
- | a job's model and effort, set on the request from its route | |
497
- | an MCP tool not on the allow list waits for the user's key, and `always` is written to the file; with nobody to ask it is refused | |
498
-
499
- The split is deliberate: a line in a prompt is a request, and the things that are cheap to check by
500
- running something are checked by running something.
105
+ `node probes/coding-eval.mjs --route sonnet/medium` runs real jobs on small tasks and scores them with hidden
106
+ tests; `--hard` runs the harder set, `--no-guides` and `--advisor` change one thing at a time. It costs money.
501
107
 
502
- ## The local model
108
+ A release is a commit on `main` that raises the version in `package.json`. The workflow runs the suite
109
+ and publishes it to npm. Probes, the coding eval and the repository layout are in
110
+ [Development](https://github.com/sumododda/sumo-agents/wiki/Development).
503
111
 
504
- One open model does the machine's bookkeeping: it routes every job and every `auto` chat turn, and it
505
- writes memory — the scribe after your turns, the dream pass over finished sessions. It is Qwen3-4B at
506
- Q4_K_M (a 2.5 GB file), run by llama.cpp's `llama-server`, downloaded once by `sumo setup` from
507
- `model.source`, `model.repo` and `model.file`. The router asks it without thinking and has an answer in
508
- about a second; the passes ask it with thinking on, capped at 1,024 tokens of thought by the server's
509
- reasoning budget, and have one in 8–15 s. Every answer is held to a JSON schema by the server's grammar,
510
- then checked by code against what you actually typed before anything is filed.
511
-
512
- It is one server, started by the first call that needs it and left running, detached, in router mode over
513
- `~/.sumo-agents/models/` with a 32k-token context. It sleeps after ten idle minutes (under 0.5 GB resident)
514
- and wakes in under a second; the first load after it starts takes up to about 15 s; it holds about 7 GB
515
- while it answers. `sumo scribe status` shows its pid, port and whether the model is loaded. `sumo setup`
516
- stops it, so a changed model file or binary takes effect. Its output goes to `~/.sumo-agents/logs/llama-server.log`.
517
-
518
- ```sh
519
- sumo config model.repo <org>/<repo> # another GGUF repository on the model source
520
- sumo config model.file <name>.gguf # the file within it — then: sumo setup
521
- sumo config scribe.model haiku # send the writer to the API instead (also sonnet, or off); dream.model likewise
522
- ```
523
-
524
- ### How it was chosen
525
-
526
- Measured on this machine (M5 Pro, 48 GB) on 2026-10-04 with the repo's own probes. The scope probe is
527
- 30 labelled turns over six conversations, 16 of them worth remembering: it asks whether the writer keeps
528
- what it should, puts each memory in the right project, proves it with a quote from what was typed, and
529
- leaves the 14 throwaway turns alone. Haiku 4.5 ran the same day for the baseline.
530
-
531
- | Writer | Remembered | Right scope | Quoted | Noise filed | Per call | Resident |
532
- |---|---|---|---|---|---|---|
533
- | **Qwen3-4B, thinking capped at 1,024 — the default** | 16/16 | 16/16 | 16/16 | 0 | 8–15 s | ~7 GB awake |
534
- | Haiku 4.5, through the API | 16/16 | 16/16 | 16/16 | 0 | 8–22 s, $0.088 for six | — |
535
- | Qwen3.5-4B, thinking capped | 16/16 | 15/16 | 16/16 | 0 | 8–13 s | 3.7 GB |
536
- | Qwen3.5-9B, thinking capped | 15/16 | 14/15 | 15/15 | 0 | 26–29 s | 6.4 GB |
537
- | Qwen3.5-9B, thinking uncapped | every call hit the token limit with no answer | | | | 137 s | |
538
- | Qwen3-4B, no thinking | 14/16 | 12/14 | 14/14 | 4 | 1–5 s | |
539
- | Qwen3.5-4B, no thinking | 14/16 | 11/14 | 14/14 | 2 | 1–4 s | |
540
- | Qwen3.5-9B, no thinking | 14/16 | 10/14 | 14/14 | 0 | 2–7 s | |
541
- | Gemma 4 E4B, no thinking | 14/16 | 13/14 | 14/14 | 0 | 2–5 s | 5.3 GB |
542
-
543
- Two things decided it. Thinking is what closes the gap to Haiku, and the cap is what makes thinking safe:
544
- uncapped, the 9B thought until the token limit on every call, and the 4B did once on a trivial prompt. And
545
- the model already here is the best of them: on the router's own grading set — 12 labelled tasks, each with
546
- the cheapest and dearest acceptable route, over three project histories — Qwen3-4B routes 11 of 12 right on
547
- each history, where Qwen3.5-4B fell to 9, 9 and 11. The larger candidates (Qwen3.6-35B-A3B, Nemotron 3 Nano
548
- 30B) are 22–25 GB files, too much to keep resident on a laptop that is also being worked on.
549
-
550
- Caveats: the probe's bundles are about 750 tokens, while real scribe bundles are 4,800 tokens at the median
551
- and 9,400 at the 90th percentile and carry the assistant's replies and related memories, so real accuracy
552
- may be lower. Every number is one run at temperature 0 of a set written by the author.
553
-
554
- ```sh
555
- node probes/scope-accuracy.mjs # the configured writer: local, $0, about 90 s
556
- node probes/scope-accuracy.mjs haiku # the same turns on the API model, about 9 cents
557
- node --disable-warning=ExperimentalWarning test/router-probes.mjs # the router's grading set
558
- ```
559
-
560
- ## What it costs
561
-
562
- - A scribe or dream pass: $0, 8–15 s in the background, no credential needed. On this machine the 224 Haiku
563
- scribe calls before the switch cost $1.86 in all, and dream's 18 cost $0.17.
564
- - The router: one local call per job, about a second once the model is loaded, $0.
565
- - The server: about 7 GB while it answers, under 0.5 GB asleep.
566
- - A web search: $0.01 and the tokens of what it found; a page read: its tokens.
567
- - The always-loaded prompt (`AGENTS.md`) is about 650 tokens, and a test keeps it there.
568
-
569
- When a local call fails (server down, a bundle past the 32k context, an answer cut off) and a credential is in
570
- the environment, the pass is retried once on Haiku; both calls are in the ledger, and `sumo scribe stats` sums
571
- it per kind. `sumo config scribe.model haiku` (or `sonnet`) sends the writer straight to the API; `off` makes
572
- the chat model save directly. `dream.model` is the same.
573
-
574
- ## Tests
575
-
576
- ```sh
577
- npm test # offline suite, no real model calls (recorded answers)
578
- node probes/scope-accuracy.mjs # live: the local writer on 30 labelled turns, $0 (`haiku` to compare, about 9 cents)
579
- node --disable-warning=ExperimentalWarning probes/coding-eval.mjs # live: eight small jobs, scored by what they never saw
580
- ```
581
-
582
- ### The coding eval
583
-
584
- Whether a change to the loop, the prompts, the tools or the router makes jobs come out right more often, or
585
- cheaper, is not something the offline suite can say. `probes/coding-eval.mjs` runs eight small jobs for real
586
- — six for a worker, two for a scout — each a project that fits on a page, written out fresh, handed over as
587
- a brief, and scored by what the job never saw: hidden tests against the tree it left, the files it was told
588
- not to touch, and for a scout the line its report was asked to end on. Each task has a way to be done wrong
589
- that still looks done: `quantity || 1` passes the test given and counts a quantity of 0 as one; a bug whose
590
- tests already pass can be left in place under a DONE; the test can be edited to agree with the code.
591
-
592
- ```sh
593
- node --disable-warning=ExperimentalWarning probes/coding-eval.mjs --list
594
- node --disable-warning=ExperimentalWarning probes/coding-eval.mjs # routed by the router
595
- node --disable-warning=ExperimentalWarning probes/coding-eval.mjs --route sonnet/medium # one model for all: only the harness varies
596
- node --disable-warning=ExperimentalWarning probes/coding-eval.mjs --task cart-quantity --runs 3 --json before.json
597
- ```
598
-
599
- It prints a line per job — `PASS`, `FAIL`, or `FALSE DONE` for a job that said DONE and was wrong, which is
600
- counted apart because nobody looks again at a DONE — with its route, requests, time and cost, and a line that
601
- sums them. It needs a credential and the router model, runs in a home of its own, and reads the real one only
602
- for which models are on. Run it before a change and after, on a pinned route, and more than once: one run of
603
- eight tasks is a small sample.
604
-
605
- The eval is itself under test (`test/coding-eval.test.mjs`): stand-in agents run the tasks through the real
606
- job machinery, and the score must tell them apart — one that does each task right passes all eight, and the
607
- sloppy fix, the idle DONE, the edited test and the report with one name too many do not.
608
-
609
- ## Layout
610
-
611
- ```
612
- AGENTS.md the system prompt of the chat, and the only always-loaded instructions
613
- prompts/ agent.md — the system prompt of a job; scribe.md, dream.md — the memory passes
614
- guides/ fix · feature · review ride in when your words ask for them; memory · projects · workflows · delegation are read on demand
615
- bin/sumo.mjs src/ the `sumo` CLI — Node, the Anthropic SDK, SQLite full-text search
616
- src/loop.mjs the loop every conversation runs in; src/tools.mjs the shell and editor with their policy
617
- src/web.mjs the web: the API's search and fetch tools, and how their blocks are read
618
- src/mcp.mjs the MCP servers: their configuration, the two transports, the allow list, `sumo mcp`
619
- src/chat.mjs the chat; src/hooks.mjs the session policy as events
620
- src/ui.mjs the chat's screen (Ink, loaded only by `sumo chat`); src/editor.mjs the box you type in
621
- spinner.txt the words the chat shows while it works — edit them; logo.txt the lion in the empty chat
622
- src/route.mjs how a job's model and effort are chosen: the router, and the stats
623
- src/local-server.mjs the one llama-server behind the router and the memory passes; src/model.mjs the calls to it and to the API
624
- test/ probes/ the suite, and the live measurements: memory scope, routing, and the coding eval
625
- docs/ADR-runtime.md the decision to own the runtime, and the plan it followed; docs/PLAN.md the original design
626
- ```
112
+ MIT licensed.