@tangle-network/agent-bench 0.8.15 → 0.8.16
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +4 -0
- package/HARNESS.md +109 -335
- package/package.json +5 -5
- package/src/agent-graphs-gen2.mts +0 -523
- package/src/agent-graphs-gen3.mts +0 -660
- package/src/agent-graphs-improve/offline-seams.mts +0 -128
- package/src/agent-graphs-improve.mts +0 -747
package/CHANGELOG.md
CHANGED
package/HARNESS.md
CHANGED
|
@@ -1,345 +1,119 @@
|
|
|
1
|
-
# bench
|
|
2
|
-
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
repeated runs on a persistent, checkable task family, the deployed policy's verifier-graded
|
|
15
|
-
**multi-objective** score (correct · fast · secure · cheap, each its own deployable checker)
|
|
16
|
-
improves **run-over-run** at matched per-run compute, surviving a frozen-policy control, significant
|
|
17
|
-
at adequate n. That across-run slope is RSI. **The harness has NOT yet run Gate B** — see the durable
|
|
18
|
-
gap below.
|
|
19
|
-
|
|
20
|
-
What the harness measures **today is Gate A** (docs/roadmap-rsi.md — the inner GO/NO-GO for the
|
|
21
|
-
within-run adaptive-driver layer): **does any non-blind topology beat blind compute at EQUAL COMPUTE
|
|
22
|
-
(Σ rollouts × turns — `k` counts rollouts, each may be multi-turn/stateful), under a DEPLOYABLE
|
|
23
|
-
(non-oracle) selector, at significant n?** Gate A is a **narrow diagnostic** — the cost-justification
|
|
24
|
-
for parallel/adaptive topology, **NOT** the product verdict. A failed Gate A deletes within-run
|
|
25
|
-
steering only; it never touches the corpus+policy product (Gate B). The invariant is equal-COMPUTE,
|
|
26
|
-
not equal-k-on-stateless-samples.
|
|
27
|
-
|
|
28
|
-
**Terminology (one word, used consistently).** A **rollout** (≡ a "shot") is ONE agent running an
|
|
29
|
-
`AgentProfile` to completion — a full, possibly **multi-turn / stateful** trajectory. `k` counts
|
|
30
|
-
*rollouts*; **turns live *inside* a rollout**, never as separate shots. A single **stateless
|
|
31
|
-
completion** (`maxTurns=0`, `harness: null`, one model call, no persistent workspace) is the
|
|
32
|
-
*degenerate* rollout — fine as a selector **lower bound**, never the canonical unit. The HumanEval
|
|
33
|
-
probe (`bench/src/humaneval-gate.mts`) uses exactly that degenerate shape — it calls the router
|
|
34
|
-
directly and does **not** route through `AgentProfile` / the sandbox / the keystone — so its numbers
|
|
35
|
-
are the **no-self-correction lower bound** on the selector, distinct from the rollout-based keystone
|
|
36
|
-
gate above. Bridge it to the product by running the same arms with real rollouts (an `AgentProfile`
|
|
37
|
-
through `runAgentRounds`), dialing `maxTurns`.
|
|
38
|
-
|
|
39
|
-
Two things to keep straight: today's judges grade a single
|
|
40
|
-
*correctness* scalar (the multi-objective vector is the open contract, architecture.md §6), and every
|
|
41
|
-
number below is single-objective + within-run — read them as Gate-A diagnostics, not Gate-B results.
|
|
42
|
-
- Within-run STEER (verify-and-revise family) **LOSES** (rung-0, n=40: blind 37.5% →
|
|
43
|
-
random@3 60.0% → refineGepa@3 45.0%; the earlier +20pp was confounded compute).
|
|
44
|
-
- On the COMMITTED finsearch corpus, the self-consistency selector also **loses**:
|
|
45
|
-
selector@k − random@k = **−8.2pp** (n=51). So "pick the consensus among k identical-ish
|
|
46
|
-
attempts" does not beat a random draw here.
|
|
47
|
-
- **VERIFIER-GROUNDED selection is the one selector that wins** — but ONLY where a domain has
|
|
48
|
-
WITHIN-TASK graded variance. Proven POSITIVE on HumanEval (deployable-checker, binary, n=50:
|
|
49
|
-
verifier−sc = +12.0pp CI[+4,+22]). The continuous-reward generalization (`selector.ts`
|
|
50
|
-
`summarizeVerifierSelector`, `corpus-replay --selector=verifier`) ranks k attempts by their
|
|
51
|
-
stored deployable-checker `score` and reports selector(=best-of-k) vs random(=mean-of-k) with a
|
|
52
|
-
paired bootstrap CI. **aec-bench is structurally DEAD for it** (n=12 gpt-4.1, **0/12 random +
|
|
53
|
-
1/12 diverse tasks have any within-task score spread**): closed-form engineering calcs score
|
|
54
|
-
deterministically w.r.t. sampling — across-task difficulty (33% resolve) but ~0 within-task
|
|
55
|
-
selection headroom. The gpt-5 null (oracle 2.5%) was a worker artifact (no JSON emission);
|
|
56
|
-
gpt-4.1 fixes the band (mean 36.1%) but the selector gate stays flat. **commit0 is the right
|
|
57
|
-
Layer-1 domain** (different impls pass different test subsets → real within-task spread).
|
|
58
|
-
- **UNTESTED (still Gate A):** parallel **DIVERSE strategies** (different reasoning paths,
|
|
59
|
-
`directives.ts` → `DIVERSE_STRATEGY_LENSES` / `composeStrategies`) @k vs blind sample(n=k). A
|
|
60
|
-
distinct family from what rung-0 falsified — the open *within-run* question, and what runProgram's
|
|
61
|
-
`parallel` is built to deploy.
|
|
62
|
-
- **UNBUILT (Gate B):** the across-run policy-improvement curve on a multi-objective task stream.
|
|
63
|
-
No harness runs it yet; it is the durable next step, not a corpus-replay over the existing
|
|
64
|
-
single-objective records.
|
|
65
|
-
|
|
66
|
-
## The run archive (where results LIVE)
|
|
67
|
-
|
|
68
|
-
Every run's full artifact is committed under `agent-lab/runs/<date>/` — self-describing JSON
|
|
69
|
-
(models + config + per-task cells + gate verdicts with CIs), portable to any repo without
|
|
70
|
-
this codebase. `agent-lab/runs/RUNS.md` is the index mapping artifacts → verdicts → the
|
|
71
|
-
findings gist. **Set `OUT=runs/<date>/<name>.json` (in agent-lab) on every run — never `/tmp`**
|
|
72
|
-
(a reboot erases ramdisk; ~20 runs nearly died there once).
|
|
73
|
-
|
|
74
|
-
## Run observability — never hand-grep a journal again
|
|
75
|
-
|
|
76
|
-
Every supervisor cell writes `run-report.json` + `run-report.md` next to its artifacts, and
|
|
77
|
-
appends a headline block to the run log, automatically: `outer-loop.mts` and both
|
|
78
|
-
`run-experiment.mts` paths call `writeRunReportSafe` after each cell and `reportRound` at
|
|
79
|
-
the gate. Source: `swe-arena/run-report.mts` (pure `computeRunReport` + an I/O collector).
|
|
80
|
-
|
|
81
|
-
What it answers, all deterministic, all from artifacts already on disk (`journal.jsonl`,
|
|
82
|
-
`workers/*.ndjson`, `workers/*.inbox.ndjson`, `state.json`, `result.json`, `judge.json`,
|
|
83
|
-
the ledger, `driver.log`, the delivered patch, the opencode session store):
|
|
84
|
-
|
|
85
|
-
- **orchestration** — workers spawned/settled/cancelled, **steer count with a per-worker
|
|
86
|
-
breakdown**, spawn waves + sizes, max concurrency, respawns, repeated labels, delegation
|
|
87
|
-
depth, time-to-first-spawn, supervisor wall, idle wall (zero live workers), worker
|
|
88
|
-
utilization (Σ worker wall ÷ supervisor wall).
|
|
89
|
-
- **decision quality** — settled by status/verdict, accepted vs rejected vs empty-pass,
|
|
90
|
-
evidence→respawn sequences vs blind respawns, review actions, evidence bytes returned.
|
|
91
|
-
- **economics** — tokens + USD by role (brain from journal `metered`; workers from journal
|
|
92
|
-
`settled` spend plus the opencode session join), per-worker rows, cost per accepted patch,
|
|
93
|
-
worker wall distribution.
|
|
94
|
-
- **outcome** — supervisor status/verdict/delivered, judge verdict + score (from `judge.json`
|
|
95
|
-
or the matching ledger row, with provenance), verify gate, patch file/line/test-touch stats.
|
|
96
|
-
|
|
97
|
-
Every metric is UNAVAILABLE-aware: a missing artifact yields `{"unavailable": "<reason>"}`
|
|
98
|
-
and renders as `unavailable — <reason>`, never `0`. A supervisor that steered nobody and a
|
|
99
|
-
supervisor whose worker logs were lost are visually different, because they have driven
|
|
100
|
-
opposite conclusions about the same architecture.
|
|
101
|
-
|
|
102
|
-
Standalone / backfill (use `--report-dir` when the run directory must stay READ-ONLY):
|
|
103
|
-
```
|
|
104
|
-
tsx src/swe-arena/run-report.mts <cellDir> [--patch p.diff] [--ledger ledger.jsonl] [--log run.log] [--report-dir out/]
|
|
105
|
-
tsx src/swe-arena/run-report.mts --round <outDir> [--report-dir out/]
|
|
106
|
-
```
|
|
1
|
+
# agent-bench: supported integration harness
|
|
2
|
+
|
|
3
|
+
`bench/` exists to prove that the published Runtime and Eval contracts can execute real benchmark adapters. It is not the research archive and it is not where successive experiment generations should accumulate.
|
|
4
|
+
|
|
5
|
+
## Repository ownership
|
|
6
|
+
|
|
7
|
+
| Repository | Owns |
|
|
8
|
+
|---|---|
|
|
9
|
+
| **agent-runtime** | exact execution, reusable benchmark adapters, packed-consumer checks, one full-fidelity integration fixture |
|
|
10
|
+
| **discovery** | research questions, preregistrations, acceptance criteria, negative results, and decisions about what is worth testing |
|
|
11
|
+
| **discovery-lab** | paid campaigns, upstream method reproductions, long-horizon hill climbs, immutable receipts, and result archives |
|
|
12
|
+
|
|
13
|
+
A benchmark implementation may begin here while it is becoming a reusable adapter. Once the question is “does method X improve benchmark Y?”, the campaign belongs in Discovery Lab.
|
|
107
14
|
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
worker accounting. The two are complementary; run-report prints the traces command for the
|
|
114
|
-
same cell rather than re-deriving session-level facts.
|
|
115
|
-
|
|
116
|
-
## Data flow (the whole experiment in one line)
|
|
117
|
-
`rollout (worker → answer) → adapter.judge (valid?) → CORPUS RunRecord (k attempts, output+valid each) → corpus-replay --selector (pick WITHOUT the judge) → corpus-report CI → gate verdict`
|
|
118
|
-
The expensive part (rollouts) produces a **reusable corpus**; selection + stats are free
|
|
119
|
-
and offline (zero new rollouts, zero judge calls).
|
|
120
|
-
|
|
121
|
-
## THE CANONICAL SUITE (2026-06-10) — the published path; start here
|
|
122
|
-
|
|
123
|
-
The optimization layer ships from the package; bench scripts compose it. A domain = an
|
|
124
|
-
`Environment` (5 hooks); a strategy = how budget is spent to beat its check; `runBenchmark`
|
|
125
|
-
returns per-strategy means + the per-task LOSSES table + the (score,$) Pareto frontier.
|
|
126
|
-
Promotion is the package gate (`promotionGate` — seeded paired bootstrap, evidence floor,
|
|
127
|
-
two modes: **superiority** and **non-inferiority** = score CI low > −tolerance AND cost
|
|
128
|
-
savings CI low > 0, the "same quality, cheaper" gate; verdicts carry paired Δlatency).
|
|
129
|
-
Authoring is `authorStrategy` (named `fallbackModel` retry). **Funnel-alignment law**: the
|
|
130
|
-
search-side champion tie-band must be no stricter than the gate's tolerance (under
|
|
131
|
-
`OBJECTIVE=cost` it defaults to it). Endurance envs on the evolve runner: `CHECKPOINT=path`
|
|
132
|
-
(phase ledger + resume — a killed run re-pays ONE phase), `GYM_RECREATE='docker …'`
|
|
133
|
-
(recreate the container at phase boundaries — the wedge killer). Observability:
|
|
134
|
-
`createWaterfallCollector` (every spawn billed+timed) + `anytimeReport` (TTT / shots-to-
|
|
135
|
-
target / COCO ERT / hill-climb AUC per satisficing target). Models policy: cheap router
|
|
136
|
-
models only (defaults `deepseek-v4-pro`/`deepseek-v4-flash`; compressor = flash with
|
|
137
|
-
`gpt-4o-mini` fallback) — never CC models; every verdict banner + artifact is
|
|
138
|
-
self-describing (models + config).
|
|
139
|
-
|
|
140
|
-
| entry point | what it answers | one-liner |
|
|
15
|
+
## Evidence levels
|
|
16
|
+
|
|
17
|
+
Use these labels literally. Do not promote one level into another in prose.
|
|
18
|
+
|
|
19
|
+
| Level | What it establishes | Canonical path |
|
|
141
20
|
|---|---|---|
|
|
142
|
-
| **
|
|
143
|
-
|
|
|
144
|
-
|
|
|
145
|
-
|
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
so N concurrent runs use N containers (`-p 8007:8005`, `-p 8008:8005`, …) instead of
|
|
161
|
-
serializing on one wedge-prone gym. Bring-up check: `agent-lab/domains/lane-probe.mts`. Cross-cutting laws baked into the suite: keep-best checkpoint scoring
|
|
162
|
-
(final-state scoring is biased −6–8pp), equal compute via the conserved pool, the analyst
|
|
163
|
-
is firewalled (trace-only), costs are real (router usage → `{usd, ms, tokens}`).
|
|
164
|
-
|
|
165
|
-
### The QUEUED runs for the test fleet (wired, one command each, unrun)
|
|
166
|
-
1. **Relevance-primed corpus A/B** — `PRIME_MODE=relevance K_FACTS=2 N=16 HOLDOUT=4` (the read-side design that survived the naive-priming negative).
|
|
167
|
-
2. ~~Strategy tournament at power~~ — RAN (n=24, budget 4, ×3 configs): HOLD verdicts; the cost-frontier finding ×3 + the funnel-alignment law came out of these. Live ledger: `.evolve/current.json` + the findings gist.
|
|
168
|
-
3. **Commit0 at real budget** — `BUDGET=3 INNER_TURNS=12 N=3` sample-vs-refine on the hard domain.
|
|
169
|
-
4. **Cross-domain replication** — blocked on sourcing the csm/hr gym containers (`EOPS_SPLIT` is wired).
|
|
170
|
-
|
|
171
|
-
## Commands (the standalone tools — each its own `main`)
|
|
172
|
-
the gate + measurement tools:
|
|
173
|
-
corpus-replay.mts --selector: selector@k vs random@k vs oracle@k over a corpus (THE offline gate)
|
|
174
|
-
corpus-report.mts paired-bootstrap CI + Benjamini-Hochberg over corpora
|
|
175
|
-
gate-cli.mts the recursive diverse-vs-blind gate through `runGate` (Supervisor)
|
|
176
|
-
run-benchmarks-cli.mts runBenchmarks: any subset of the ADAPTERS registry × model/harness cells, one combined ranked report (#420)
|
|
177
|
-
commit0-env-run.mts the HARD domain through `runBenchmark` (the optimization suite)
|
|
178
|
-
terminal-compare.ts Terminal-Bench compare (own main)
|
|
179
|
-
pnpm verify:pier zero-model failure/pass Pier controls through a separate verifier
|
|
180
|
-
swe-arena/run-report.mts deterministic per-cell run report (steers, waves, idle, cost by role) + round rollup
|
|
181
|
-
unit tests (the only fully-green, cred-free runnable surface besides offline replay):
|
|
182
|
-
node --test --import tsx src/{selector,refine-loop}.test.mts
|
|
183
|
-
tsx src/gate.test.mts # offline plumbing test (no creds)
|
|
184
|
-
|
|
185
|
-
## Run the GATE — today, zero creds (it already runs)
|
|
186
|
-
```
|
|
187
|
-
cd bench
|
|
188
|
-
pnpm gate # = corpus-replay.mts corpus/finsearch.jsonl --selector
|
|
189
|
-
tsx src/corpus-replay.mts corpus/finsearch.jsonl --selector --condition=refine # other arms
|
|
190
|
-
tsx src/corpus-replay.mts <corpus.jsonl> --selector=verifier # GRADED domains: rank k by deployable-checker score
|
|
191
|
-
pnpm gate-report # paired-bootstrap CI + BH-FDR
|
|
21
|
+
| **contract proof** | packages install; identities, budgets, callbacks, resume, and receipts have the expected shape | root `pnpm verify:official-optimizers`, `pnpm verify:primeintellect`, `pnpm verify:bench` |
|
|
22
|
+
| **evaluator proof** | the benchmark's own evaluator can distinguish known fail/pass artifacts in the exact environment | adapter preflight and gold/self-check |
|
|
23
|
+
| **reproduction proof** | an upstream method is run at a pinned revision on its claimed benchmark under a matched protocol | Discovery Lab reproduction manifest and runner |
|
|
24
|
+
| **value proof** | the integrated method beats the preregistered baseline on frozen evidence with uncertainty and complete cost accounting | Discovery Lab result receipt |
|
|
25
|
+
| **production proof** | a promoted artifact transfers to real traffic under a canary or controlled rollout | product repository / platform telemetry |
|
|
26
|
+
|
|
27
|
+
A localization score, output-shape check, LLM quality judge, or toy deterministic reward can be useful for development. None is a substitute for the benchmark's outcome evaluator.
|
|
28
|
+
|
|
29
|
+
## Supported commands
|
|
30
|
+
|
|
31
|
+
### Package and integration contracts
|
|
32
|
+
|
|
33
|
+
From the repository root:
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
pnpm verify:bench
|
|
37
|
+
pnpm verify:official-optimizers
|
|
38
|
+
pnpm verify:primeintellect
|
|
192
39
|
```
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
## Run the DIVERSE-vs-blind gate THROUGH the keystone (the recursive runtime, live)
|
|
40
|
+
|
|
41
|
+
`verify:official-optimizers` exercises the official Optimize Anything bridge, engine identities, equal input budgets, resume compatibility, candidate callbacks, accounting, and package provenance. Its deterministic candidate improvement is deliberately a fixture. It does **not** reproduce the published GEPA or Omni benchmark numbers.
|
|
42
|
+
|
|
43
|
+
### Bounded benchmark matrix
|
|
44
|
+
|
|
45
|
+
From `bench/`:
|
|
46
|
+
|
|
47
|
+
```bash
|
|
48
|
+
pnpm run run-benchmarks
|
|
204
49
|
```
|
|
50
|
+
|
|
51
|
+
`src/run-benchmarks-cli.mts` runs a selected subset of registered adapters across explicit agent cells. Each adapter owns task loading, output extraction, preflight, and judging. A missing dependency or failed gold self-check makes the benchmark unavailable; it never becomes a zero score.
|
|
52
|
+
|
|
53
|
+
Use `LOOP_ATTEMPTS=N` only when the benchmark's own visible feedback is allowed to enter later attempts. Hidden or gold material must remain outside the agent context.
|
|
54
|
+
|
|
55
|
+
### Full-fidelity improvement fixture
|
|
56
|
+
|
|
57
|
+
```bash
|
|
205
58
|
cd bench
|
|
206
|
-
|
|
207
|
-
BENCH=enterpriseops-gym EOPS_FIXTURES=1 N=20 K=4 pnpm gate-cli
|
|
208
|
-
```
|
|
209
|
-
`gate-cli.mts` → `runGate` (`src/gate.ts`): a `Persona` + the generic
|
|
210
|
-
`fanout` combinator over the budget-conserving `Supervisor`. Blind = K identical children, diverse
|
|
211
|
-
= K distinct strategy directives — equal-k by construction (conserved pool), proven by
|
|
212
|
-
`equalKOnCost`. The DEPLOYABLE selector is the benchmark's OWN `adapter.judge` (each child solves
|
|
213
|
-
via the router, is graded by the runnable checker, and that `BenchScore` is the child's verdict
|
|
214
|
-
`defaultSelectWinner` ranks on — selector ≠ oracle/LLM-judge). Pick a deployable-checker bench
|
|
215
|
-
(enterpriseops-gym / swe-bench / terminal-bench), NOT finsearchcomp (LLM-judge → not deployable).
|
|
216
|
-
Offline plumbing test (no creds): `tsx src/gate.test.mts`. The gate runs through the SAME recursive
|
|
217
|
-
atom every personified loop uses.
|
|
218
|
-
|
|
219
|
-
## "Supervisor" (iterate/decompose) vs blind — through the PUBLISHED suite
|
|
220
|
-
The supervisor-vs-blind gate is NOT a bespoke harness: it is `runBenchmark([sample, refine, …])`
|
|
221
|
-
over an Environment. blind = `sample` (best-of-k); "supervisor" = `refine`/`sampleThenRefine`
|
|
222
|
-
(depth: attempt→firewalled-analyst-steer→retry — *"a multi-agent team is just a Strategy whose driver
|
|
223
|
-
spawns several agents"*). Equal compute by the substrate's CONSERVED budget; the deployable check is
|
|
224
|
-
the Environment's `score`; the can't-fake-the-check firewall is built in. Run it on the HARD real
|
|
225
|
-
domain via `commit0-env-run.mts` (above) or the toy `strategy-demo.mts` (offline). The LLM
|
|
226
|
-
agent-driver (an LLM that itself decides spawns via the coordination MCP) is the SEPARATE product
|
|
227
|
-
path — `atom-mcp-e2e.mts` / `atom-commit0.mts` — not a strategy. Evolve any strategy on a frozen
|
|
228
|
-
holdout with `runStrategyEvolution`.
|
|
229
|
-
|
|
230
|
-
## Generate a fresh corpus + gate it
|
|
231
|
-
The rollout generators now live with their domains: the recursive gate
|
|
232
|
-
(`gate-cli.mts`) and the optimization-suite env runs (`commit0-env-run.mts`,
|
|
233
|
-
`research-gate.mts` for the off-sandbox RAG baseline) each append corpus `RunRecord`s. Gate any
|
|
234
|
-
written corpus offline with the selector:
|
|
235
|
-
```
|
|
236
|
-
tsx src/corpus-replay.mts <corpus.jsonl> --selector
|
|
237
|
-
```
|
|
238
|
-
(hotpotqa is cheap + deterministic-judge but near-ceiling/weak-signal; simpleqa similar;
|
|
239
|
-
finsearchcomp is the strong-signal domain but needs the sandbox/local-web worker.)
|
|
240
|
-
|
|
241
|
-
## Optimize the strategy/prompt (so the gate tests BEST-effort, not strawman)
|
|
242
|
-
Strategy-space search is the package's `runStrategyEvolution` (the optimization suite); the diverse
|
|
243
|
-
lenses (`directives.ts`) layer on top of the shared base directive consumed by the gate arms.
|
|
244
|
-
|
|
245
|
-
## Workers (the rollout substrate)
|
|
246
|
-
The gate solves each child via the router and grades it with the benchmark's own
|
|
247
|
-
deployable `adapter.judge`; `research-gate.mts` is the off-sandbox retrieve→answer baseline
|
|
248
|
-
(`SEARCH=<provider>` selects the web-search arm). The steer text lives in `directives.ts`, NOT in the
|
|
249
|
-
worker (the worker is substrate). A strategy is a prompt PREFIX; the judge is unchanged.
|
|
250
|
-
|
|
251
|
-
## Adapters (benchmarks/) — honest state (the code wins over this line; verified 2026-06-04)
|
|
252
|
-
The code-benches share `benchmarks/_harness.ts` (stage artifact → run the bench's OWN evaluator
|
|
253
|
-
in a `.venv`/Docker subprocess → parse its JSON report → `{resolved,score}`). No per-adapter
|
|
254
|
-
copy of the process/venv/Docker/temp/report plumbing; commit0+appworld also share its
|
|
255
|
-
stdin-piping runner (`runVenvScriptStdin`).
|
|
256
|
-
Published-package consumers set `AGENT_BENCH_PYTHON` to the absolute path of an interpreter containing the benchmark dependencies.
|
|
257
|
-
If unset, source checkouts keep using `bench/.venv/bin/python`.
|
|
258
|
-
SWE-bench callers that need a hard evaluation deadline construct `createSweBenchAdapter({ timeoutMs })`; evaluator errors and incomplete or ambiguous reports throw instead of scoring zero.
|
|
259
|
-
Callers that must inspect the exact per-task image after scoring pass `cacheLevel: 'instance'`; the default remains `'env'`.
|
|
260
|
-
Callers retain the complete official evaluator tree plus raw process output with `captureEvaluatorArtifacts: ({ taskId, attemptSequence }) => ({ destination })`; the returned score (or `StagedJudgeError`) carries a receipt with every file hash and a whole-tree hash.
|
|
261
|
-
- **Real, runnable with ZERO extra deps:** finsearchcomp (GitHub dataset + fixtures + LLM judge — the gate bench), hotpotqa + simpleqa + frames (HF/web QA + F1/LLM judge; `*_FIXTURES=1` offline), **ragbench**, **crag**, **nomiracl**, **open-rag-bench**, **t2-ragbench** (SOTA RAG/knowledge benchmarks with committed fixtures and deterministic answer/relevance judges; live mode reads explicit `*_DATA_FILE` JSON/JSONL exports), **aec-bench** (real GitHub task tree + fixtures; judge = the task's own `tests/verify.py` over python3 stdlib — **deterministic, graded per-field partial credit, no Docker, no LLM** → the candidate non-oracle correctable-middle-band bench for the open gate).
|
|
262
|
-
- **Real code, needs an external harness/tools to run (fail loud with the exact install/Docker fix; never a fabricated score):** swe-bench + terminal-bench (`bench/.venv` + Docker), **commit0** (ISOLATED `bench/.venv-commit0` via `python3 -m venv bench/.venv-commit0 && bench/.venv-commit0/bin/pip install commit0 datasets` — its deps conflict with the shared `.venv`; override dir with `COMMIT0_VENV` — plus Docker; judge = official pytest harness, graded (passed+xfail)/total; the rollout prompt stages in-box (clones `commit-0/<repo>` @ `base_commit`, emits `git diff`); `COMMIT0_FIXTURES=1` for offline listing), **programbench** (`pip install programbench` + Docker on linux/amd64 + HF blobs; judge = official cleanroom eval, graded passed/total; `PROGRAMBENCH_FIXTURES=1` offline), **appworld** (`pip install appworld` + `appworld install` + `appworld download data`; judge = AppWorld's own `world.evaluate()`, graded passes/num_tests — NO committed fixture: task data exists only after `download data`, so loadTasks fails loud rather than fabricate a task), **dabstep** (`DABSTEP_DIR=/path/to/EnvCommons/DABStep` with the released `dataset.csv`, `splits/*.txt`, `files/*`, and `grade.py`; judge delegates to official `grade.py`; `DABSTEP_FIXTURES=1` only tests adapter plumbing and does not fabricate benchmark scores), **webarena-verified** (`WEBARENA_VERIFIED_DIR=/path/to/webarena-verified`; judge delegates to official `eval-tasks` over a run output directory), **tau2-bench** (`TAU2_BENCH_DIR=/path/to/tau2-bench`; judge recomputes tau2 trajectory rewards), **tau3-banking** (`TAU3_BENCH_DIR=/path/to/tau2-bench`; default domain `banking_knowledge`; judge recomputes tau trajectory rewards through the upstream tau3 package), **agentbench** DBBench subset (`AGENTBENCH_DIR=/path/to/AgentBench`; exact-match deterministic label judge), **bfcl** deterministic function-call subset (`BFCL_DIR=/path/to/gorilla/berkeley-function-call-leaderboard`; loads official BFCL JSONL + `possible_answer`; score = structured call/argument match, not the full BFCL leaderboard evaluator), **toollm** API-selection subset (`TOOLBENCH_DIR=/path/to/ToolBench`; score = recall of ToolBench `relevant APIs` labels, resolved only when the worker emits the requested structured JSON call list; official ToolEval pass rate remains LLM-judged/stochastic), **finresearchbench** (`FINRESEARCHBENCH_DATA_FILE=/path/to/export.jsonl`; rows must carry official `judge_system_prompt` + `judge_prompt_template`; no self-authored live judge), mind2web, cad-design + cadbench + cadgenbench (openscad/blender/build123d), **mcad** (openscad + xvfb-run; 10 dimensioned mechanical parts adopted from the MIT text-to-cad benchmark set; judge = OpenSCAD compile then pure-TS mesh assertions — bbox, closed-form volume bands, body count, and ray-parity point-in-solid probes at the spec's pinned hole coordinates; all 10 tasks carry a verified gold, and the must-reject direction is tested: one missing bore of four fails on its named probe), **mcad-cq** (the SAME 10 parts, tasks and spec assertions, but the worker writes a Python **CadQuery** script that must export BOTH `part.step` and an ASCII `part.stl` — which closes the format deviation `mcad-tasks.ts` documents, since upstream asks for STEP and OpenSCAD cannot emit it; the geometry engine is imported from `mcad-bench.ts`, not copied, and STEP delivery is one EXTRA scored check `stepEmitted` = `part.step` starts with `ISO-10303-21`; 120 s deadline per script, all 10 tasks carry a verified CadQuery gold at score 1.0, and four must-rejects are tested — one bore deleted, no STL, a binary STL, and the STEP export dropped. Needs an ISOLATED `bench/.venv-cadquery`: `cd bench && uv venv --python 3.12 .venv-cadquery && uv pip install --python .venv-cadquery/bin/python 'cadquery==2.4.0' 'numpy<2'` — both pins are load-bearing, cadquery-ocp 7.7.2 has no cp313 wheel and cadquery 2.4.0's pinned nptyping 2.0.1 reads the numpy-1.x-only `np.bool8`; override with the absolute path `MCAD_CQ_PYTHON`. The judge RUNS the worker's Python on the host, same trust level as `mcad` running arbitrary OpenSCAD).
|
|
263
|
-
- **goldArtifact:** aec-bench returns the task's real `golden_pass.md` (verify-judge works fully offline). commit0 / programbench / appworld return `undefined` — the oracle is a git ref / stripped source / engine-bundled solution, not a portable string; judge correctness is proven by a real solve through the harness, not a synthetic gold (documented + fail-loud, not a fake).
|
|
264
|
-
- **Absent (not built):** swe-gym, swe-bench-multimodal, and the rest of the survey set.
|
|
265
|
-
Every unbuilt/scaffold adapter fails LOUD (throws with the integration step) rather than faking a score — no silent zeros in any corpus. Offline fixture tests: `benchmarks/{aec-bench,commit0,programbench,appworld,rag-benchmarks}.test.mts` (`tsx --test`).
|
|
266
|
-
|
|
267
|
-
## Pier candidate bridge
|
|
268
|
-
|
|
269
|
-
`pier_agents.tangle_candidate:TangleCandidateAgent` is the reusable Pier custom-agent path for frozen Tangle candidates.
|
|
270
|
-
`executePreparedPierCandidate()` is the only public entry point; its private staging step writes the runtime's execution-plan and materialization-receipt bytes verbatim.
|
|
271
|
-
The Python bridge rechecks those bytes and their signed task, candidate, profile, repository, instruction, and workspace identities before launch.
|
|
272
|
-
It rejects a raw candidate bundle, never projects an `AgentProfile`, and leaves task isolation, patch transfer, verification, retries, and result storage to Pier.
|
|
273
|
-
|
|
274
|
-
The adapter fails the trial when any prepared identity drifts, when the task checkout or immutable OCI image differs from the signed identity, or when the candidate exits nonzero or exceeds its signed deadline.
|
|
275
|
-
Candidate code runs as an unprivileged numeric user, while evaluator inputs and timeout evidence remain root-owned.
|
|
276
|
-
It never accepts candidate-authored token, cost, or trace receipts; `executePreparedPierCandidate()` uses the runtime's atomic execution path and reconciles the protected `TraceStore` with the model-gateway ledger before returning a gradable receipt.
|
|
277
|
-
The prepared object contains no credentials.
|
|
278
|
-
The runtime passes model and trace bindings only to the trusted executor request.
|
|
279
|
-
The Pier launcher inherits their values through its protected process environment and passes only `${NAME}` references on the command line, so credentials never enter prepared bytes, CLI arguments, or job files.
|
|
280
|
-
The executor builds fresh task, candidate, and profile trees from the request's exact verified file bytes; prepared staging directories are never launch authority.
|
|
281
|
-
Isolated memory and knowledge-bearing candidates currently fail closed until Pier has executor-owned mount and after-state capture.
|
|
282
|
-
The signed wall deadline is a hard stop: the runtime aborts, Pier kills the process tree, and the executor acknowledges process and container death.
|
|
283
|
-
The signed tool-step count is a post-run validity check over protected traces, not a pre-tool stop; generic black-box Pier processes cannot honestly prevent step N+1.
|
|
284
|
-
The executable zero-model fixture is `fixtures/pier-agent/`; run it against the R360 Pier checkout with `PIER_REPO=/path/to/pier pnpm verify:pier`.
|
|
285
|
-
That command runs a no-change candidate that must score 0/1, proves a fresh evaluator process can kill a persisted child and remove its real Docker container, and runs a known-good candidate that must score 1/1.
|
|
286
|
-
It then checks that each official result and exact task patch is bound into its own runtime receipt with zero model usage.
|
|
287
|
-
|
|
288
|
-
For a real frozen candidate, use `FilePierCandidateTrialController`, append `agentArgs` and `attemptArgs`, and pass each executor-only `evaluatorEnv` entry through Pier's evaluator-owned environment mechanism.
|
|
289
|
-
The launch callback's second argument carries the signed runtime `request`, protected `traceStore`, cancellation `signal`, and absolute `deadlineAtMs`; production launchers must use that context rather than reconstructing it.
|
|
290
|
-
The controller sends secrets to its supervisor over a pipe, while its durable files contain only process and Docker-project identities:
|
|
291
|
-
The supervisor never inherits `process.env`; non-default Docker connections use a stable `dockerConnection.id` plus the exact environment injected into both Pier and cleanup.
|
|
292
|
-
Fresh recovery workers must reconstruct that same named connection; `terminate-pier-trial.mts` selects only the comma-delimited variables named by `PIER_DOCKER_ENV_NAMES` when `PIER_DOCKER_CONNECTION_ID` is set.
|
|
293
|
-
`jobName` must be unique per prepared execution; the controller atomically reserves that job directory so recovery can remove only containers owned by that execution.
|
|
294
|
-
|
|
295
|
-
```ts
|
|
296
|
-
const controller = new FilePierCandidateTrialController({
|
|
297
|
-
directory: '/var/lib/tangle/pier-control',
|
|
298
|
-
launch: (staged, { request }) => {
|
|
299
|
-
const jobName = request.executionId
|
|
300
|
-
return {
|
|
301
|
-
command: 'uv',
|
|
302
|
-
args: ['run', 'pier', 'run', ...staged.agentArgs, ...staged.attemptArgs],
|
|
303
|
-
cwd: pierCheckout,
|
|
304
|
-
env: { ...evaluatorEnvironment, ...staged.evaluatorEnv },
|
|
305
|
-
jobsDirectory,
|
|
306
|
-
jobName,
|
|
307
|
-
readResult: () => readOfficialPierResult(jobsDirectory, jobName),
|
|
308
|
-
}
|
|
309
|
-
},
|
|
310
|
-
})
|
|
311
|
-
|
|
312
|
-
const result = await executePreparedPierCandidate({
|
|
313
|
-
prepared,
|
|
314
|
-
directory: '/sealed/candidate',
|
|
315
|
-
pierVersion: '0.3.0',
|
|
316
|
-
traceStore,
|
|
317
|
-
claimStore,
|
|
318
|
-
outputArtifacts,
|
|
319
|
-
grader,
|
|
320
|
-
controller,
|
|
321
|
-
})
|
|
59
|
+
pnpm tsx src/swe-self-improve.mts
|
|
322
60
|
```
|
|
323
61
|
|
|
324
|
-
|
|
325
|
-
On a signed deadline, external abort, or recovery by another evaluator process, the adapter waits for `terminateAndWait()` to acknowledge both process exit and container removal before it returns control to the runtime.
|
|
62
|
+
This is the canonical real-task Runtime fixture:
|
|
326
63
|
|
|
327
|
-
|
|
328
|
-
|
|
329
|
-
|
|
64
|
+
- SWE-bench Verified instances;
|
|
65
|
+
- repository state as the produced artifact;
|
|
66
|
+
- the official Docker judge outside the candidate agent;
|
|
67
|
+
- explicit train, selection, and frozen final-test partitions;
|
|
68
|
+
- Runtime's `improve()` boundary and complete cost receipts.
|
|
330
69
|
|
|
331
|
-
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
|
|
70
|
+
It proves the integrated execution path can support a real value campaign. A paid powered result still belongs in Discovery Lab.
|
|
71
|
+
|
|
72
|
+
### Offline diagnostics
|
|
73
|
+
|
|
74
|
+
```bash
|
|
75
|
+
cd bench
|
|
76
|
+
pnpm gate
|
|
77
|
+
pnpm gate-report
|
|
335
78
|
```
|
|
336
|
-
|
|
337
|
-
|
|
338
|
-
|
|
339
|
-
|
|
340
|
-
|
|
341
|
-
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
|
|
345
|
-
|
|
79
|
+
|
|
80
|
+
`corpus-replay.mts` and `corpus-report.mts` are retained for offline analysis of already-recorded attempts. They do not create new evidence and must not be presented as a live reproduction unless the source corpus itself has a pinned, independently verified receipt.
|
|
81
|
+
|
|
82
|
+
## Upstream-method reproduction requirements
|
|
83
|
+
|
|
84
|
+
A reproduction record is incomplete unless it binds all of the following:
|
|
85
|
+
|
|
86
|
+
1. upstream repository, package, and exact revision;
|
|
87
|
+
2. the upstream claim and benchmark protocol being reproduced;
|
|
88
|
+
3. adapter and evaluator identity;
|
|
89
|
+
4. dataset revision and split commitments;
|
|
90
|
+
5. baseline and treatment definitions;
|
|
91
|
+
6. models, harnesses, temperatures, seeds, and complete budgets;
|
|
92
|
+
7. optimization, candidate-execution, judge, and final-test costs separately;
|
|
93
|
+
8. raw outputs, evaluator reports, and terminal failure classes;
|
|
94
|
+
9. a parity criterion declared before the run;
|
|
95
|
+
10. an immutable receipt linking every artifact above.
|
|
96
|
+
|
|
97
|
+
Exact score equality is usually the wrong parity criterion for stochastic systems. Prefer a preregistered band, direction-of-effect, rank ordering, or confidence interval that is strong enough to detect an integration error.
|
|
98
|
+
|
|
99
|
+
## Method map
|
|
100
|
+
|
|
101
|
+
- **GEPA** — optimizer over explicit candidate surfaces and evaluation feedback.
|
|
102
|
+
- **AutoResearch / Prime Agent** — agentic search over an editable research surface. Treat the agent, tools, and external evaluator as separate identities.
|
|
103
|
+
- **Meta-Harness** — search over harness or orchestration behavior; preserve the same outcome evaluator.
|
|
104
|
+
- **Omni** — phase-one portfolio search followed by a fresh phase-two optimizer seeded from the best phase-one artifact. The matched phase-one budgets and the phase boundary are part of the protocol.
|
|
105
|
+
- **Trace analysts** — evidence producers. Measure finding quality against labeled traces before using findings to steer search.
|
|
106
|
+
- **Prime Agent RLM and DSPy RLM** — alternative analyst/context engines, not optimization methods by themselves. Compare them on the same trace questions, evidence requirements, context budgets, and downstream decisions.
|
|
107
|
+
|
|
108
|
+
Do not put all of these into one undifferentiated “intelligence” arm. They intervene at different points in the causal chain.
|
|
109
|
+
|
|
110
|
+
## Admission rule for new bench code
|
|
111
|
+
|
|
112
|
+
A new file under `bench/src` must be one of:
|
|
113
|
+
|
|
114
|
+
- a reusable benchmark adapter;
|
|
115
|
+
- a shared execution/evaluator primitive used by more than one adapter;
|
|
116
|
+
- a package-consumer or evaluator calibration test;
|
|
117
|
+
- one canonical full-fidelity fixture that exercises a public Runtime contract.
|
|
118
|
+
|
|
119
|
+
A one-off campaign, generation-N optimizer script, bespoke dashboard, or historical result belongs in Discovery Lab. If an older file has no package script, no importer, and no unique reusable primitive, delete it rather than adding another index entry.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@tangle-network/agent-bench",
|
|
3
|
-
"version": "0.8.
|
|
3
|
+
"version": "0.8.16",
|
|
4
4
|
"type": "module",
|
|
5
5
|
"description": "Benchmark adapters and execution for agent-runtime across coding, tool-use, RAG, memory, browser, and terminal tasks.",
|
|
6
6
|
"repository": {
|
|
@@ -25,11 +25,11 @@
|
|
|
25
25
|
}
|
|
26
26
|
},
|
|
27
27
|
"dependencies": {
|
|
28
|
-
"@tangle-network/agent-eval": ">=0.
|
|
28
|
+
"@tangle-network/agent-eval": ">=0.149.0 <0.150.0",
|
|
29
29
|
"@tangle-network/agent-interface": "^1.1.0",
|
|
30
|
-
"@tangle-network/agent-knowledge": "^8.0.
|
|
31
|
-
"@tangle-network/sandbox": ">=0.
|
|
32
|
-
"@tangle-network/agent-runtime": "^0.
|
|
30
|
+
"@tangle-network/agent-knowledge": "^8.0.10",
|
|
31
|
+
"@tangle-network/sandbox": ">=0.29.0 <0.30.0",
|
|
32
|
+
"@tangle-network/agent-runtime": "^0.142.0"
|
|
33
33
|
},
|
|
34
34
|
"devDependencies": {
|
|
35
35
|
"@arethetypeswrong/cli": "0.18.5",
|