tare-cli 0.2.0__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- tare/__init__.py +1 -0
- tare/agents.py +672 -0
- tare/calibrate.py +98 -0
- tare/capsule.py +246 -0
- tare/cli.py +312 -0
- tare/cliff.py +183 -0
- tare/dashboard.html +269 -0
- tare/dashboard.py +216 -0
- tare/fake.py +183 -0
- tare/journal.py +35 -0
- tare/judge.py +108 -0
- tare/overview.html +78 -0
- tare/probe.py +273 -0
- tare/recipe.py +102 -0
- tare/room.py +67 -0
- tare/swap.py +219 -0
- tare/trail.py +203 -0
- tare/usage.py +237 -0
- tare_cli-0.2.0.dist-info/METADATA +505 -0
- tare_cli-0.2.0.dist-info/RECORD +23 -0
- tare_cli-0.2.0.dist-info/WHEEL +4 -0
- tare_cli-0.2.0.dist-info/entry_points.txt +2 -0
- tare_cli-0.2.0.dist-info/licenses/LICENSE +21 -0
|
@@ -0,0 +1,505 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: tare-cli
|
|
3
|
+
Version: 0.2.0
|
|
4
|
+
Summary: Prove your coding agent starts without your personal setup before you measure it.
|
|
5
|
+
Project-URL: Homepage, https://github.com/a-ratz/tare
|
|
6
|
+
Project-URL: Repository, https://github.com/a-ratz/tare
|
|
7
|
+
Project-URL: Issues, https://github.com/a-ratz/tare/issues
|
|
8
|
+
Project-URL: Concept, https://github.com/a-ratz/tare/blob/main/CONCEPT.md
|
|
9
|
+
Author: Andre Ratzenberger
|
|
10
|
+
License-Expression: MIT
|
|
11
|
+
License-File: LICENSE
|
|
12
|
+
Keywords: agent-evaluation,ai-agents,claude-code,codex,coding-agents,evals,llm-evaluation,sandbox
|
|
13
|
+
Classifier: Development Status :: 3 - Alpha
|
|
14
|
+
Classifier: Environment :: Console
|
|
15
|
+
Classifier: Intended Audience :: Developers
|
|
16
|
+
Classifier: Operating System :: POSIX :: Linux
|
|
17
|
+
Classifier: Programming Language :: Python :: 3
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
21
|
+
Classifier: Topic :: Software Development :: Testing
|
|
22
|
+
Requires-Python: >=3.11
|
|
23
|
+
Description-Content-Type: text/markdown
|
|
24
|
+
|
|
25
|
+
<div align="center">
|
|
26
|
+
<picture>
|
|
27
|
+
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/a-ratz/tare/main/.github/logo-dark.svg">
|
|
28
|
+
<source media="(prefers-color-scheme: light)" srcset="https://raw.githubusercontent.com/a-ratz/tare/main/.github/logo-light.svg">
|
|
29
|
+
<img alt="tare" src="https://raw.githubusercontent.com/a-ratz/tare/main/.github/logo-light.svg" width="260">
|
|
30
|
+
</picture>
|
|
31
|
+
|
|
32
|
+
<p>Prove your coding agent starts without your personal setup before you measure it.</p>
|
|
33
|
+
</div>
|
|
34
|
+
|
|
35
|
+
<div align="center">
|
|
36
|
+
|
|
37
|
+
[![License: MIT][license-shield]][license-url]
|
|
38
|
+
[![PyPI][version-shield]][version-url]
|
|
39
|
+
[![Python 3.11+][python-shield]][python-url]
|
|
40
|
+
[![Linux | WSL][platform-shield]][platform-url]
|
|
41
|
+
[![Claude Code | Codex | Pi | Antigravity][agent-shield]][agent-url]
|
|
42
|
+
|
|
43
|
+
</div>
|
|
44
|
+
|
|
45
|
+
<div align="center">
|
|
46
|
+
<a href="https://github.com/a-ratz/tare#install">Install</a> ·
|
|
47
|
+
<a href="#quick-start">Quick start</a> ·
|
|
48
|
+
<a href="#features">Features</a> ·
|
|
49
|
+
<a href="#how-it-works">How it works</a> ·
|
|
50
|
+
<a href="https://github.com/a-ratz/tare/blob/main/CONCEPT.md">Concept</a> ·
|
|
51
|
+
<a href="https://github.com/a-ratz/tare/issues/new?template=bug_report.md">Report Bug</a>
|
|
52
|
+
</div>
|
|
53
|
+
|
|
54
|
+
<br>
|
|
55
|
+
|
|
56
|
+
---
|
|
57
|
+
|
|
58
|
+
## Why tare?
|
|
59
|
+
|
|
60
|
+
You test a skill by comparing agent runs with the skill against runs without it. On your own
|
|
61
|
+
machine, both runs also carry your personal setup: your global instructions, memories, installed
|
|
62
|
+
skills, MCP servers and connected accounts. The comparison then measures your machine, not the
|
|
63
|
+
skill.
|
|
64
|
+
|
|
65
|
+
A separate config directory (`CLAUDE_CONFIG_DIR` for Claude Code, `CODEX_HOME` for Codex) does not
|
|
66
|
+
fix this. With Claude Code, the login alone brings your connected accounts (mail, calendar,
|
|
67
|
+
documents), your account's skills and your email address into the session. Evaluation frameworks
|
|
68
|
+
that drive agent CLIs, such as UiPath's coder_eval, leave it to you to provide a clean
|
|
69
|
+
environment, and none of them checks it.
|
|
70
|
+
|
|
71
|
+
tare starts the agent in a clean environment and proves that it is clean before every run. It
|
|
72
|
+
works with four coding agents on Linux and WSL: Claude Code, Codex, Pi
|
|
73
|
+
([`@earendil-works/pi-coding-agent`](https://www.npmjs.com/package/@earendil-works/pi-coding-agent))
|
|
74
|
+
and Google's Antigravity CLI (`agy`).
|
|
75
|
+
|
|
76
|
+
## The words tare uses
|
|
77
|
+
|
|
78
|
+
The name comes from weighing. You tare a scale, which sets it to zero, before you weigh
|
|
79
|
+
something. tare does the same for an agent before you measure it.
|
|
80
|
+
|
|
81
|
+
| Word | Meaning |
|
|
82
|
+
|---|---|
|
|
83
|
+
| room | An isolated environment for one agent run. It has an empty home directory, a fresh copy of your login (the file in which the agent keeps your sign-in) and only four basic environment variables from your shell (an allowlist), plus its own `HOME` and `PATH`. Your project appears at `/work`. tare builds the room with [bubblewrap](https://github.com/containers/bubblewrap). |
|
|
84
|
+
| probe | The test that tare runs before an agent starts. It makes no paid model calls. The agent talks to a fake model server on your machine, which records what the agent sends. |
|
|
85
|
+
| reading | The result of a probe, one of three. A clean room reads `tare: 0.00`, the zeroed scale. A room with leaks reads, for example, `tare: 4 leaks`. A probe that could not see your setup reads `tare: not proven (blind)`. There are no values in between. |
|
|
86
|
+
| leak | Something of your personal setup that reaches the room, for example your global instructions. The reading names each leak and the file it came from. |
|
|
87
|
+
| declared | Something of yours that the room holds on purpose, for example the copy of your login. The reading lists it, but it is not a leak and does not stop the run. |
|
|
88
|
+
| dirty twin | The same agent run in your real setup, outside the room. It is part of the probe and proves that the probe can see your setup at all. |
|
|
89
|
+
|
|
90
|
+
## Features
|
|
91
|
+
|
|
92
|
+
- **Checks the room before every run.** The agent starts only when the probe reads
|
|
93
|
+
`tare: 0.00`. Otherwise you see every leak with the file it came from.
|
|
94
|
+
- **Measures what the model receives.** tare does not ask the agent about its context. It records
|
|
95
|
+
the request that the agent sends to its model. An agent asked about its own context can refuse
|
|
96
|
+
or be wrong.
|
|
97
|
+
- **Tells you when it cannot see.** tare looks at your files to learn what your setup holds, for
|
|
98
|
+
example a global instructions file or installed skills. If the dirty twin's request lacks one of
|
|
99
|
+
these, the probe could not see it. A clean room would then prove nothing. tare runs the dirty
|
|
100
|
+
twin once more, and if that kind of context is still missing, the reading is
|
|
101
|
+
`tare: not proven (blind)`.
|
|
102
|
+
- **Keeps your files out of reach.** Your home directory, the credentials of your other tools
|
|
103
|
+
and the secret environment variables of your shell do not exist inside the room.
|
|
104
|
+
- **Keeps your own login working.** Each room gets a fresh copy of your login, which tare deletes
|
|
105
|
+
after the run. Claude Code may replace its login token when it renews it, which could log you
|
|
106
|
+
out outside tare. That has not been tested, so tare refuses to start when the copy would need
|
|
107
|
+
renewal within the next hour. Start the agent once outside tare, which renews
|
|
108
|
+
your login, and try again.
|
|
109
|
+
- **Passes all your arguments to the agent.** Interactive or one-shot, the agent runs as it does
|
|
110
|
+
outside tare.
|
|
111
|
+
- **Compares skills, plugins and models.** `tare calibrate` starts each side (an agent with its
|
|
112
|
+
arguments) several times from scratch, each in its own room. It reports how often each side
|
|
113
|
+
passes a check, a shell command such as your test suite.
|
|
114
|
+
- **Finds out whether the workspace (the run's copy of your project) or the agent caused a
|
|
115
|
+
failure.** `tare swap` lets two agents
|
|
116
|
+
continue each other's unfinished work and compares the results.
|
|
117
|
+
- **Looks for the step where a failed run went wrong** (experimental). `tare cliff` restarts a
|
|
118
|
+
failed run from its saved steps (one per tool call) and names the step after which it no
|
|
119
|
+
longer succeeds.
|
|
120
|
+
|
|
121
|
+
## When to use
|
|
122
|
+
|
|
123
|
+
| Use tare when | Look elsewhere when |
|
|
124
|
+
|---|---|
|
|
125
|
+
| you compare agent runs with and without a skill, plugin or prompt | you need protection from a hostile agent, because the room has network access and a working login |
|
|
126
|
+
| you benchmark agents on your own machine | you are on macOS (a room is in progress, not released yet) |
|
|
127
|
+
| you want a fresh-machine run without a fresh machine | you log in with an API key only (untested) |
|
|
128
|
+
| an agent failed a task and you want to know which step lost it | you want a single run explained without running the agent again, because Cliff runs it many times |
|
|
129
|
+
|
|
130
|
+
## Install
|
|
131
|
+
|
|
132
|
+
| Requirement | Notes |
|
|
133
|
+
|---|---|
|
|
134
|
+
| Linux or WSL2 | tare builds the room with user namespaces |
|
|
135
|
+
| [bubblewrap](https://github.com/containers/bubblewrap) | `sudo apt install bubblewrap` |
|
|
136
|
+
| at least one of the four agents | on `PATH` and logged in. Codex must resolve to a path under `/usr` for now, because tare mounts Codex's npm package and Node.js from there. If `which codex` points elsewhere, for example into nvm, install it with the system's npm (`sudo npm install -g @openai/codex`) and put `/usr/bin` first on `PATH`. The other agents can live anywhere on `PATH`. |
|
|
137
|
+
| Google Chrome, only for `tare judge` | `google-chrome` on `PATH`, installed under `/usr` or `/opt`. `tare judge` scores web pages and takes their screenshots with it. |
|
|
138
|
+
| [uv](https://docs.astral.sh/uv/) or [pipx](https://pipx.pypa.io/) | Python 3.11 or newer |
|
|
139
|
+
|
|
140
|
+
Install tare from PyPI. The package is `tare-cli`, the command is `tare`:
|
|
141
|
+
|
|
142
|
+
```bash
|
|
143
|
+
uv tool install tare-cli # or: pipx install tare-cli
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
To work on tare itself, clone the repository and run it from the checkout with `uv run tare ...`:
|
|
147
|
+
|
|
148
|
+
```bash
|
|
149
|
+
git clone https://github.com/a-ratz/tare.git && cd tare
|
|
150
|
+
uv run tare --version
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
## Quick start
|
|
154
|
+
|
|
155
|
+
With tare installed:
|
|
156
|
+
|
|
157
|
+
```bash
|
|
158
|
+
tare probe claude
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
```text
|
|
162
|
+
tare probe · claude 2.1.289 · /home/me/tare
|
|
163
|
+
control dirty twin shows: instructions, home path, mcp, skills, plugins, email, reach
|
|
164
|
+
leaks none
|
|
165
|
+
declared
|
|
166
|
+
email account email subscription login
|
|
167
|
+
credentials /home/tare/.claude-config/.credentials.json copy of the login, needed by the CLI
|
|
168
|
+
tare: 0.00
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
This is the output of a real run, with only the home path shortened. Line by line:
|
|
172
|
+
|
|
173
|
+
- `control` lists the kinds of context that the dirty twin found in your real setup. The probe
|
|
174
|
+
could therefore have seen each of them in the room. Connected accounts count under `mcp`,
|
|
175
|
+
because they reach the agent as MCP tools. `home path` means that your real home path appears
|
|
176
|
+
in the agent's request, for example in the path of a memory file. `reach` comes from a plain
|
|
177
|
+
script that looks for your files. It finds them outside the room, which shows that the script
|
|
178
|
+
works. So when it finds nothing inside the room, that result can be trusted.
|
|
179
|
+
- `leaks none` says that none of these reached the room. Inside the room, tare starts Claude Code
|
|
180
|
+
with flags that keep connected accounts and account skills out.
|
|
181
|
+
- `declared` lists the two things that the room holds on purpose. With a subscription login, your
|
|
182
|
+
account email arrives with the login, and tare cannot remove it without removing the login. It
|
|
183
|
+
is one line of text that names you, not instructions, memories or tools. So tare reports it
|
|
184
|
+
and lets the run start. The room also holds a copy of your login, because the agent needs one
|
|
185
|
+
to run.
|
|
186
|
+
|
|
187
|
+
For the dirty twin, the probe starts your real agent with your real setup, so your hooks and MCP
|
|
188
|
+
servers start too. The Antigravity CLI writes into its config directory on every run, so tare
|
|
189
|
+
lets its dirty twin write into a temporary overlay that is discarded after the run. Your
|
|
190
|
+
`~/.gemini` stays unchanged.
|
|
191
|
+
|
|
192
|
+
`tare claude` then starts Claude Code in that room. `tare codex`, `tare pi` and `tare agy` do
|
|
193
|
+
the same for the other agents.
|
|
194
|
+
|
|
195
|
+
## Usage
|
|
196
|
+
|
|
197
|
+
```bash
|
|
198
|
+
tare probe claude # probe the room: tare: 0.00, the leaks and their sources, or blind
|
|
199
|
+
tare claude # probe, then start Claude Code in the room
|
|
200
|
+
tare claude -- -p "fix the tests" # arguments after -- go to claude
|
|
201
|
+
tare claude --yolo # skip the agent's permission prompts
|
|
202
|
+
tare claude --allow-dirty # start even though the reading is not tare: 0.00
|
|
203
|
+
tare claude --project ../other # another project directory (default: the current one)
|
|
204
|
+
|
|
205
|
+
tare probe codex # the same for Codex, Pi and the Antigravity CLI
|
|
206
|
+
tare codex -- exec -m MODEL "..." # one-shot Codex run: arguments after -- go to codex
|
|
207
|
+
tare pi # Pi in a room
|
|
208
|
+
tare agy # the Antigravity CLI in a room
|
|
209
|
+
tare prices update # fetch the price table for costs an agent does not report
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
`tare claude`, `tare codex`, `tare pi` and `tare agy` mount your project at `/work` directly. The
|
|
213
|
+
agent changes your real files, as it would outside the room. `--yolo` adds the agent's own flag
|
|
214
|
+
for skipping permission prompts. Pi has no such prompts. The commands under
|
|
215
|
+
[Measuring agents in rooms](#measuring-agents-in-rooms) work on copies instead.
|
|
216
|
+
|
|
217
|
+
`tare probe` exits with 0 on `tare: 0.00` and with 1 otherwise. `tare claude` and the other agent
|
|
218
|
+
commands refuse to start with exit code 1 unless the reading is `tare: 0.00` or you pass
|
|
219
|
+
`--allow-dirty`. When they start, they return the agent's own exit code.
|
|
220
|
+
|
|
221
|
+
The room starts with the agent's defaults, not with your settings. Codex, for example, runs its
|
|
222
|
+
default model unless you pass `-m`. Set the model explicitly when you compare runs.
|
|
223
|
+
|
|
224
|
+
When a room is not clean, the reading names each leak and where it came from. The excerpt below
|
|
225
|
+
shows a room that was given copies of real settings on purpose, with the home path shortened.
|
|
226
|
+
|
|
227
|
+
```text
|
|
228
|
+
leaks
|
|
229
|
+
instructions global instructions /home/me/.claude/CLAUDE.md
|
|
230
|
+
home path /home/me/ your files
|
|
231
|
+
skills 1 skill /home/me/.claude/skills
|
|
232
|
+
env SPIKE_API_KEY inherited environment
|
|
233
|
+
tare: 4 leaks
|
|
234
|
+
```
|
|
235
|
+
|
|
236
|
+
## Measuring agents in rooms
|
|
237
|
+
|
|
238
|
+
The commands in this section start many agent runs. Each run works on its own copy of your
|
|
239
|
+
project in its own room. tare scores each run with a **check**: any shell command that you pass
|
|
240
|
+
with `--check`. tare runs it in the finished workspace, and exit code 0 means that the run
|
|
241
|
+
passed. A **step** is one tool call of the agent, such as running a command or writing a file.
|
|
242
|
+
|
|
243
|
+
Each command saves everything in a **run directory** under `~/.local/state/tare/`. It also starts
|
|
244
|
+
a live dashboard on your machine and prints its address. Open the address in a browser to follow
|
|
245
|
+
the runs.
|
|
246
|
+
|
|
247
|
+
A pass rate comes with a **95% interval**: the range that holds the true pass rate with 95%
|
|
248
|
+
confidence. Few runs give a wide interval. Three passes out of three, for example, still allow a
|
|
249
|
+
true rate as low as 0.44.
|
|
250
|
+
|
|
251
|
+
### Calibrate: how often does each side pass?
|
|
252
|
+
|
|
253
|
+
```bash
|
|
254
|
+
tare calibrate "fix the failing test" --check "uv run pytest -q" \
|
|
255
|
+
--side "claude --model haiku" --side "codex -m gpt-6.1-sol" --runs 10
|
|
256
|
+
```
|
|
257
|
+
|
|
258
|
+
Each `--side` is an agent with its arguments. tare starts every side `--runs` times from a fresh
|
|
259
|
+
copy of the project and reports each side's pass rate with its interval. Use it to compare
|
|
260
|
+
skills, plugins or models. It also tells you whether a task suits Swap or Cliff. Swap needs a
|
|
261
|
+
task where one side mostly fails and the other mostly passes. Cliff needs a task where a fresh
|
|
262
|
+
start passes at least sometimes.
|
|
263
|
+
|
|
264
|
+
Some skills start with their own command, for example a slash command, so the sides need
|
|
265
|
+
different prompts. `--side-prompt` gives a side its own prompt. The sides are named a, b and c in
|
|
266
|
+
the order of `--side`:
|
|
267
|
+
|
|
268
|
+
```bash
|
|
269
|
+
tare calibrate --side "claude --plugin-dir plugins/a" --side "claude --plugin-dir plugins/b" \
|
|
270
|
+
--side-prompt "a=/a:ideate a todo app" --side-prompt "b=/b:ideate a todo app" --runs 5
|
|
271
|
+
```
|
|
272
|
+
|
|
273
|
+
Without `--check`, a run counts as finished when its agent exits with 0. tare keeps each finished
|
|
274
|
+
workspace, and the report shows the finished runs and their times, so that you can score the
|
|
275
|
+
results elsewhere. `--keep` keeps the workspaces when there is a check too.
|
|
276
|
+
|
|
277
|
+
### Judge: scoring results that no test can decide
|
|
278
|
+
|
|
279
|
+
Some results have no test, for example how good a web page is. A **judge** is a second agent that
|
|
280
|
+
scores the result from 0 to 100 by your **rubric**, a Markdown file that says what earns points.
|
|
281
|
+
The judge is Claude Code with Sonnet unless you choose another agent with `--judge`.
|
|
282
|
+
|
|
283
|
+
`tare judge DIR` opens the page `DIR/index.html` (another file with `--page`) in headless Google
|
|
284
|
+
Chrome inside a room of its own and takes a screenshot. Without `DIR`, it uses the current
|
|
285
|
+
directory. The judge then gets the screenshot, the page's source and the rubric in a fresh room,
|
|
286
|
+
without any agent or model names, so it cannot favour an agent or a model. `tare judge` exits
|
|
287
|
+
with 0 when the score reaches the `--threshold` value, so it works as a check:
|
|
288
|
+
|
|
289
|
+
```bash
|
|
290
|
+
tare calibrate "build the page" --check "tare judge --rubric $PWD/rubric.md --threshold 60" \
|
|
291
|
+
--side "claude --model haiku" --runs 5
|
|
292
|
+
```
|
|
293
|
+
|
|
294
|
+
A check runs in the finished workspace, so `tare judge` without `DIR` judges the page that the
|
|
295
|
+
agent built. The check does not run where you typed the command. So the rubric needs an absolute
|
|
296
|
+
path, which your shell makes from `$PWD/rubric.md`. And `tare` must be on `PATH`, or written as
|
|
297
|
+
`uv run --project <checkout> tare`.
|
|
298
|
+
|
|
299
|
+
A judge does not give the same page the same score every time. One test page scored 55, 60, 58,
|
|
300
|
+
58, 66 and 62 in six judgements. Before you set a threshold, let `tare judge-noise` score a few
|
|
301
|
+
finished pages several times each, ideally a good one and a weak one:
|
|
302
|
+
|
|
303
|
+
```bash
|
|
304
|
+
tare judge-noise good-page/ weak-page/ --rubric rubric.md --times 10 --threshold 60
|
|
305
|
+
```
|
|
306
|
+
|
|
307
|
+
Each directory holds one finished page. `judge-noise` lists every page's scores and refuses a
|
|
308
|
+
threshold that lies inside the range of a page's scores, because there the judge's noise would
|
|
309
|
+
decide between pass and fail.
|
|
310
|
+
|
|
311
|
+
### Swap: was it the workspace or the agent?
|
|
312
|
+
|
|
313
|
+
```bash
|
|
314
|
+
tare swap "fix the failing test" --check "uv run pytest -q" --a claude --a-args "--model sonnet" --b codex
|
|
315
|
+
```
|
|
316
|
+
|
|
317
|
+
Two agents, a and b, each attempt the same task once. When one of them fails, two causes are
|
|
318
|
+
possible. The agent may be the weaker one. Or its workspace may have reached a state that no
|
|
319
|
+
agent can finish. Swap separates the two causes.
|
|
320
|
+
|
|
321
|
+
Swap cuts both runs at several points. `--cuts` sets them as fractions of each run's steps,
|
|
322
|
+
where 0 is the start and 1 is the end. The default is `0,0.5,1`. At each cut, each agent continues
|
|
323
|
+
each workspace, three times by default (`--tails`). Each such continuation is a **tail**. A tail
|
|
324
|
+
gets the workspace and a **handoff**: a written summary of the steps so far, in the same form for
|
|
325
|
+
every agent. Swap then reports two numbers per cut, each with an interval:
|
|
326
|
+
|
|
327
|
+
- The **state effect** says how much better the tails do in a's workspace than in b's, which is
|
|
328
|
+
how much the state of the workspace matters.
|
|
329
|
+
- The **model effect** says how much better agent a does than agent b, in the same workspaces.
|
|
330
|
+
The model effect covers the agent CLI and its model together.
|
|
331
|
+
|
|
332
|
+
At cut 0 both workspaces are your untouched project, so the state effect there must be zero. Swap
|
|
333
|
+
checks this and calls it the **null check**. Where an agent can continue its own saved session
|
|
334
|
+
(Claude Code, Codex and Pi), Swap also lets each agent continue its own session from each cut.
|
|
335
|
+
That measures what the handoff itself costs. The report shows it per agent as a **handoff cost**
|
|
336
|
+
line: the agent's pass rate when it continues its own session, minus its pass rate from the
|
|
337
|
+
handoff, in its own workspace. A value near zero means that the handoff loses nothing.
|
|
338
|
+
|
|
339
|
+
The report below comes from tare's own test. It drives the real Claude Code CLI against two
|
|
340
|
+
scripted models: fake models that follow a fixed script, so the right verdict is known. The test
|
|
341
|
+
runs four tails per workspace and agent instead of three. Agent a is competent but trusts a note in the
|
|
342
|
+
workspace. Agent b is weak. Run b wrote a wrong note at step 2.
|
|
343
|
+
|
|
344
|
+
Each cell shows passes out of tails and their interval. Positive effects favour a's workspace or
|
|
345
|
+
agent a. The report continues with cut 1.00, left out here.
|
|
346
|
+
|
|
347
|
+
```text
|
|
348
|
+
cut 0.00: workspace a at step 0, workspace b at step 0
|
|
349
|
+
agent a agent b
|
|
350
|
+
workspace a 4/4 0.51-1.00 1/4 0.05-0.70
|
|
351
|
+
workspace b 4/4 0.51-1.00 1/4 0.05-0.70
|
|
352
|
+
state effect +0.00 [-0.35, +0.35] (positive: workspace a better)
|
|
353
|
+
model effect +0.75 [+0.28, +0.89] (positive: agent a better)
|
|
354
|
+
handoff cost a +0.00 [-0.49, +0.49] (own session 4/4 against handoff 4/4, both in its own workspace)
|
|
355
|
+
handoff cost b +0.50 [-0.14, +0.79] (own session 3/4 against handoff 1/4, both in its own workspace)
|
|
356
|
+
|
|
357
|
+
cut 0.50: workspace a at step 2, workspace b at step 2
|
|
358
|
+
agent a agent b
|
|
359
|
+
workspace a 4/4 0.51-1.00 1/4 0.05-0.70
|
|
360
|
+
workspace b 0/4 0.00-0.49 2/4 0.15-0.85
|
|
361
|
+
state effect +0.38 [-0.03, +0.66] (positive: workspace a better)
|
|
362
|
+
model effect +0.12 [-0.25, +0.44] (positive: agent a better)
|
|
363
|
+
handoff cost a +0.00 [-0.49, +0.49] (own session 4/4 against handoff 4/4, both in its own workspace)
|
|
364
|
+
handoff cost b -0.25 [-0.66, +0.32] (own session 1/4 against handoff 2/4, both in its own workspace)
|
|
365
|
+
|
|
366
|
+
null check passed: no state effect at cut 0, where both workspaces are the untouched project
|
|
367
|
+
verdict Blame passes from the model to the workspace between cut 0.00 and cut 0.50. The intervals behind this verdict include zero. More tails would settle it.
|
|
368
|
+
```
|
|
369
|
+
|
|
370
|
+
At the start, the model effect explains the difference. Agent a passes, agent b mostly fails.
|
|
371
|
+
After the wrong note, a's workspace still works and b's does not, even for the competent agent
|
|
372
|
+
a. The verdict says so.
|
|
373
|
+
|
|
374
|
+
### Cliff: where did a failed run go wrong?
|
|
375
|
+
|
|
376
|
+
```bash
|
|
377
|
+
tare cliff claude "fix the failing test" --check "uv run pytest -q" -- --model sonnet
|
|
378
|
+
```
|
|
379
|
+
|
|
380
|
+
A failed run is a chain of steps. Somewhere in that chain, the run may have taken a turn that it
|
|
381
|
+
could not recover from. Cliff looks for that step, the **cliff**. Before it the run could still
|
|
382
|
+
succeed, after it the run no longer does.
|
|
383
|
+
|
|
384
|
+
> **Experimental.** Cliff finds the known step in tare's scripted test. On real runs it has
|
|
385
|
+
> never blamed a step that was not the cause. But it has not yet found a real cliff either,
|
|
386
|
+
> because neither of two real test rounds had a failure that sat in one step. The rounds were a
|
|
387
|
+
> [data migration](https://github.com/a-ratz/tare/blob/main/experiments/migration/RESULTS.md) and a [web page](https://github.com/a-ratz/tare/blob/main/experiments/html/RESULTS.md).
|
|
388
|
+
|
|
389
|
+
tare runs the task once in a room and saves the workspace and the conversation after every step.
|
|
390
|
+
If the check fails, Cliff starts tails from saved steps in fresh rooms. Tails from the beginning
|
|
391
|
+
are the **baseline**. They show how often the task succeeds at all.
|
|
392
|
+
|
|
393
|
+
Cliff first runs three baseline tails and three tails from the last step. A step counts as good
|
|
394
|
+
while its tails pass at least half as often as the baseline. Cliff then halves the range between
|
|
395
|
+
the last good step and the first bad one, three tails at a time. When the two are neighbours, it
|
|
396
|
+
adds tails to both until their intervals no longer overlap, or until the budget (`--budget`,
|
|
397
|
+
default 30 tails) is spent. Everything after `--` goes to every agent run, so pin the model there.
|
|
398
|
+
|
|
399
|
+
Cliff can also end without naming a step. When even the baseline almost never passes, the task is
|
|
400
|
+
too hard for the model, and Cliff reports a model gap. It does so only when the upper end of the
|
|
401
|
+
baseline's interval lies below `--gap-below` (default 0.2). When tails from the last step pass at
|
|
402
|
+
least half as often as the baseline, the run was unlucky, not lost, and Cliff says that.
|
|
403
|
+
|
|
404
|
+
The report below comes from tare's scripted test, with a scripted model that writes the wrong
|
|
405
|
+
answer into a note at step 4. Steps that Cliff did not need are missing from the table, and `<`
|
|
406
|
+
marks the cliff. `monotone` says whether a step after the cliff passes again.
|
|
407
|
+
|
|
408
|
+
```text
|
|
409
|
+
step what the step did tails pass rate 95% interval
|
|
410
|
+
0 start 3 3 1.00 0.44-1.00
|
|
411
|
+
3 Bash echo hi > scratch.txt 6 5 0.83 0.44-0.97
|
|
412
|
+
4 Bash echo answer=41 > notes.txt 6 0 0.00 0.00-0.39 <
|
|
413
|
+
6 Bash echo 41 > answer.txt 3 0 0.00 0.00-0.56
|
|
414
|
+
|
|
415
|
+
verdict The run became lost at step 4.
|
|
416
|
+
at step 4: Bash echo answer=41 > notes.txt
|
|
417
|
+
changed notes.txt
|
|
418
|
+
monotone yes: no step after the cliff passes again
|
|
419
|
+
tails 18 of a budget of 30
|
|
420
|
+
```
|
|
421
|
+
|
|
422
|
+
Cliff works with all four agents (`tare cliff pi ...`). Claude Code, Codex and Pi continue their
|
|
423
|
+
own saved session. The Antigravity CLI stores its sessions in a format that tare cannot cut, so
|
|
424
|
+
Cliff continues it with a handoff, as Swap does. Each tail is a real agent run, so a search costs
|
|
425
|
+
as much as its tails.
|
|
426
|
+
|
|
427
|
+
### Tokens and cost
|
|
428
|
+
|
|
429
|
+
Every Calibrate, Swap and Cliff report ends with the tokens and the cost of each side, of the
|
|
430
|
+
check (a judge agent costs tokens too) and of the whole run:
|
|
431
|
+
|
|
432
|
+
```text
|
|
433
|
+
usage a claude --model haiku: 1 run, 42.3k in (34.8k cached), 352 out, $0.020. Subscription login, so the cost is notional
|
|
434
|
+
b codex -m gpt-6.1-sol: 1 run, 42.1k in (38.5k cached), 129 out, about $0.012 (estimated from LiteLLM prices fetched 2026-10-05). Subscription login, so the cost is notional
|
|
435
|
+
check (the judge): 2 runs, 102.0k in (92.7k cached), 471 out, $0.061
|
|
436
|
+
total: 4 runs, 186.4k in (166.0k cached), 952 out, about $0.093 (estimated from LiteLLM prices fetched 2026-10-05)
|
|
437
|
+
```
|
|
438
|
+
|
|
439
|
+
Claude Code and Pi report their cost themselves. Codex and the Antigravity CLI report only
|
|
440
|
+
tokens, so tare estimates their cost from a price table, and only when it knows the model: pass
|
|
441
|
+
it with `-m` or `--model`. `tare prices update` fetches LiteLLM's public price table, and
|
|
442
|
+
`~/.config/tare/prices.json` overrides single models with your own prices (in dollars per million
|
|
443
|
+
tokens, with a source and a date). With a subscription login the cost is what the same tokens
|
|
444
|
+
would cost through the API, so the report calls it notional. The dashboard shows the same
|
|
445
|
+
totals while the run goes on, and the run directory keeps them in `usage.json`.
|
|
446
|
+
|
|
447
|
+
### Watch it live, repeat it
|
|
448
|
+
|
|
449
|
+
The live dashboard of `tare calibrate`, `tare swap` and `tare cliff` runs at
|
|
450
|
+
`http://localhost:8777/` unless that port is taken. It shows the setup of the run, the probe
|
|
451
|
+
readings, each run step by step, every tail with what its agent is doing, the results as they
|
|
452
|
+
come in, and the final report.
|
|
453
|
+
|
|
454
|
+
```bash
|
|
455
|
+
tare watch ~/.local/state/tare/cliff/myproject-20261005-142000 # any run, live or finished
|
|
456
|
+
tare watch ~/.local/state/tare/calibrate/* # several runs: one row each
|
|
457
|
+
tare rerun ~/.local/state/tare/cliff/myproject-20261005-142000 # the same run again
|
|
458
|
+
```
|
|
459
|
+
|
|
460
|
+
Every run directory holds a `recipe.json` with the command, the task and the check, a fingerprint
|
|
461
|
+
of the project, the agents with their versions and arguments, and the settings. `tare rerun`
|
|
462
|
+
repeats the run in a new directory with the same settings and says what has changed since and
|
|
463
|
+
what the repeated run used.
|
|
464
|
+
Agent runs are not deterministic, so a repeated run can still come out differently.
|
|
465
|
+
|
|
466
|
+
## How it works
|
|
467
|
+
|
|
468
|
+
A probe has three parts, and none of them asks the agent anything. First, the real agent runs
|
|
469
|
+
inside the room against the fake model server, which keeps the request that it receives. That
|
|
470
|
+
request is the context the agent was given. Second, the dirty twin runs the same agent in your
|
|
471
|
+
real setup, so tare knows that it can see your setup at all. Third, a plain script inside the
|
|
472
|
+
room looks for your files and for secret environment variables.
|
|
473
|
+
|
|
474
|
+
→ [Concept: the design and the measurements behind it](https://github.com/a-ratz/tare/blob/main/CONCEPT.md)
|
|
475
|
+
→ [The dirty-twin experiments that decided the probe design: Claude Code](https://github.com/a-ratz/tare/blob/main/experiments/dirty-twin/RESULTS.md), [Codex](https://github.com/a-ratz/tare/blob/main/experiments/dirty-twin-codex/RESULTS.md), [Pi](https://github.com/a-ratz/tare/blob/main/experiments/dirty-twin-pi/RESULTS.md), [Antigravity CLI](https://github.com/a-ratz/tare/blob/main/experiments/dirty-twin-agy/RESULTS.md)
|
|
476
|
+
|
|
477
|
+
## Roadmap
|
|
478
|
+
|
|
479
|
+
A room for macOS, built on Seatbelt, is in progress ([epic #85](https://github.com/a-ratz/tare/issues/85)).
|
|
480
|
+
Until it is released, tare runs on Linux and WSL.
|
|
481
|
+
|
|
482
|
+
## Contributing
|
|
483
|
+
|
|
484
|
+
Every change starts as an issue. Larger work is an epic, split into features, and one epic
|
|
485
|
+
becomes one branch and one pull request. [AGENTS.md](https://github.com/a-ratz/tare/blob/main/AGENTS.md) describes the process, the checks
|
|
486
|
+
and the conventions. Run the tests with `uv run pytest`.
|
|
487
|
+
|
|
488
|
+
## License
|
|
489
|
+
|
|
490
|
+
MIT. See [LICENSE](https://github.com/a-ratz/tare/blob/main/LICENSE).
|
|
491
|
+
|
|
492
|
+
---
|
|
493
|
+
|
|
494
|
+
Crafted with [Readme Craft](https://github.com/motiful/readme-craft)
|
|
495
|
+
|
|
496
|
+
[license-shield]: https://img.shields.io/badge/license-MIT-blue.svg
|
|
497
|
+
[license-url]: https://github.com/a-ratz/tare/blob/main/LICENSE
|
|
498
|
+
[version-shield]: https://img.shields.io/pypi/v/tare-cli.svg
|
|
499
|
+
[version-url]: https://github.com/a-ratz/tare/blob/main/pyproject.toml
|
|
500
|
+
[python-shield]: https://img.shields.io/badge/python-3.11%2B-3776AB.svg
|
|
501
|
+
[python-url]: https://www.python.org/
|
|
502
|
+
[platform-shield]: https://img.shields.io/badge/platform-Linux%20%7C%20WSL-555555.svg
|
|
503
|
+
[platform-url]: #install
|
|
504
|
+
[agent-shield]: https://img.shields.io/badge/agents-Claude%20Code%20%7C%20Codex%20%7C%20Pi%20%7C%20Antigravity-D97757.svg
|
|
505
|
+
[agent-url]: #install
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
tare/__init__.py,sha256=rZuSHkVpIuRmCEmJ6b-GmiJOd9CIpJpmq_hAW5Jxvjg,67
|
|
2
|
+
tare/agents.py,sha256=Rd2YmHpnTZnNqgsnb_ERlc1yxFcULkV1M0_yRxu6C3k,31963
|
|
3
|
+
tare/calibrate.py,sha256=LnYDgZLVipqdACyDVZSNyd2RpFJrBpVn-7m0HbEg73A,4673
|
|
4
|
+
tare/capsule.py,sha256=ZSUmtYBQGi3Buvy4KmgjxELy3UyoaBU1uD7hnmeI2WE,11340
|
|
5
|
+
tare/cli.py,sha256=KkSUhdQ2zKPO-_ea5KpltRdhfyIC7hQGLjAwMIRW6KM,18484
|
|
6
|
+
tare/cliff.py,sha256=cxN6xx4t75Kqif9TYKG6qGT3V45HzR0Myg38KP7n0UQ,8633
|
|
7
|
+
tare/dashboard.html,sha256=CtX3WB1gXvvxoXocZ_WZLUmzWyjau4kJQwvht2NEQYE,19129
|
|
8
|
+
tare/dashboard.py,sha256=I-rCL30aj3FkU1zlmMI9FMiMFMfsKRXj0JuoQZf4b2s,9878
|
|
9
|
+
tare/fake.py,sha256=3U3qBd3OKcTB5f4VtAbbWdmU_8aMEmjBhA7hrXgJmWY,9690
|
|
10
|
+
tare/journal.py,sha256=XRpgGjzn5YrRRa8v49t-kKQ4vtoK1wJsy0yodfj68Cc,1091
|
|
11
|
+
tare/judge.py,sha256=o1mM5lkG9s9vi4bb8hi1CatV3t6n0sGThgYN81Cl23U,6129
|
|
12
|
+
tare/overview.html,sha256=6KrIreXi5pYMGY84VXHn_xCMWA9O2QG7jHtmUO5kNEw,3917
|
|
13
|
+
tare/probe.py,sha256=lJS4CWhncp9v_KXdLavhi_hllOOIP40PFvrtUSWnfKg,12952
|
|
14
|
+
tare/recipe.py,sha256=-LFnjTrcpfzmGjhilGWworlAxOK8c3psp_mGiLADLYM,3903
|
|
15
|
+
tare/room.py,sha256=Ls3DirJcCkGLMz6n5XFI7_-zBlYeng-OUAtoHJ3yhgU,3019
|
|
16
|
+
tare/swap.py,sha256=4HEAWY7AIHdRKTC3NqQFLIMr-J9Frh8rRnpr8mVKTqE,11397
|
|
17
|
+
tare/trail.py,sha256=vAUOPz9m1CwESTMjs8sJ5YpdwdAdtjbd-sZBC33iGsQ,9952
|
|
18
|
+
tare/usage.py,sha256=O4Od_v-ulJ-mylxY55djFLQIufn4dFVd1Zqv_m739tU,10812
|
|
19
|
+
tare_cli-0.2.0.dist-info/METADATA,sha256=4pWSDzqd7FA8dZtG6qAL3W9C1wGcWFb746sV_vA0Xw4,28628
|
|
20
|
+
tare_cli-0.2.0.dist-info/WHEEL,sha256=W3fkpkm7-wf9vBI5Z-7s0eWkeM-spu78I8Neb98DeEg,87
|
|
21
|
+
tare_cli-0.2.0.dist-info/entry_points.txt,sha256=BhoQrZYP3BRJuI7i3u3G5m50IrH6MDFcla6OL51d31U,39
|
|
22
|
+
tare_cli-0.2.0.dist-info/licenses/LICENSE,sha256=ZN4Y2sSRK1FL3-LSXp_sEfXthN-X-zLQeO7LzHeD4fY,1075
|
|
23
|
+
tare_cli-0.2.0.dist-info/RECORD,,
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Andre Ratzenberger
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|