@tangle-network/agent-bench 0.3.7 → 0.3.8

package/src/swe-arena/fixtures/run-report/factory-agent-eval-309-FSUP0.md

@@ -0,0 +1,100 @@
+# Run report — factory.agent-eval.309 [FSUP0]
+
+```
+RUN-REPORT factory.agent-eval.309 [FSUP0]
+  steers=0 (spawn→wait→respawn only; no mid-task steering)
+  waves=4 sizes=[2,1,1,1] workers=5 settled=5 cancelled=0
+  concurrency max=2 utilization=0.663 idle=15.6min (48.3%) wall=32.2min
+  respawns=3 evidence→respawn=3 blind-respawn=0 depth=1
+  accepted=3 rejected=1 empty-pass=1
+  brain=$0.179577 total=$0.179577 judge.resolved=false score=0.4333 verify=unavailable — result.json absent or has no verify_pass
+  gaps(1): verifyPass: result.json absent or has no verify_pass
+```
+
+- Cell dir: `/tmp/factory-gen0-live/rep-0/runs/factory.agent-eval.309/FSUP0`
+- Supervisor: `sup-1-8f75ee`
+- Generated: 2026-07-24T00:32:17.691Z
+
+## Orchestration
+
+| Metric | Value |
+|---|---|
+| Workers spawned | 5 |
+| Workers settled | 5 |
+| Workers cancelled | 0 |
+| **Steers (mid-task messages to live workers)** | **0** |
+| Steers delivered | 0 |
+| Outer-driver `supervisor_steer` calls | 0 |
+| Spawn waves | 4 |
+| Wave sizes | [2, 1, 1, 1] |
+| Max concurrency | 2 |
+| Respawns (spawns after first settle) | 3 |
+| Repeated labels | none |
+| Delegation depth | 1 |
+| Time to first spawn | 13.9min |
+| Supervisor wall | 32.2min |
+| Idle (zero live workers) | 15.6min (48.3%) |
+| Worker utilization (Σ worker wall ÷ supervisor wall) | 0.663 |
+
+### Steers per worker
+
+| Worker | Queued | Delivered |
+|---|---:|---:|
+| `w-0` | 0 | 0 |
+| `w-1` | 0 | 0 |
+| `w-2` | 0 | 0 |
+| `w-3` | 0 | 0 |
+| `w-4` | 0 | 0 |
+
+## Decision quality
+
+| Metric | Value |
+|---|---|
+| Settled by status | done=5 |
+| Settled verdicts | none |
+| Accepted (verify green + patch bytes) | 3 |
+| Rejected (verify red) | 1 |
+| Empty pass (green, no patch) | 1 |
+| Evidence → respawn sequences | 3 |
+| Respawn with no settled evidence in front | 0 |
+| Review actions (steers + worker questions) | 0 |
+| Worker evidence returned | 12153 bytes |
+
+## Economics
+
+| Role | Tokens in | Tokens out | USD | Source |
+|---|---:|---:|---:|---|
+| brain | 228169 | 19398 | 0.179577 | journal metered events (n=17) |
+| workers | 286763 | 69794 | 0 | journal settled spend + opencode sessions (n=5) |
+
+- Total USD: 0.179577 (source: state.json result.spentUsd — brain-priced only; worker CLI inference is unpriced (see worker token counts))
+- Cost per accepted patch: 0.059859
+- Worker wall (n=5): min 70.4s / p50 3.9min / p90 9.3min / max 9.3min / Σ 21.2min
+
+| Worker | Wall | Patch bytes | Verify passed |
+|---|---:|---:|---|
+| `w-0` | 9.3min | 21183 | false |
+| `w-1` | 4.7min | 14573 | true |
+| `w-2` | 70.4s | 0 | true |
+| `w-3` | 3.9min | 21325 | true |
+| `w-4` | 2.2min | 1016 | true |
+
+## Outcome
+
+| Metric | Value |
+|---|---|
+| Supervisor status | completed |
+| Supervisor verdict | delivered |
+| Delivered | true |
+| Judge resolved | false |
+| Judge score | 0.4333 |
+| Judge passed / total | 13 / 30 |
+| Judge source | /tmp/factory-gen0-live/ledger.jsonl (ledger row) |
+| Verify gate | pass=unavailable — result.json absent or has no verify_pass rc=unavailable — result.json absent or has no verify_rc |
+| Patch | 5 file(s), +939/-0, test files touched: src/capability-headroom.test.ts, src/paired-arms.test.ts |
+
+## Gaps
+
+- verifyPass: result.json absent or has no verify_pass
+
+> Harness-session view of the same cell (model calls, stuck loops, tool errors): `npx --yes @tangle-network/traces@latest analyze --harness opencode --cwd <worker-clone-cwd>`

package/src/swe-arena/fixtures/run-report/gen3-rollup.json

@@ -0,0 +1,551 @@
+{
+ "schema": "swe-arena/run-report-rollup@1",
+ "cells": 51,
+ "steersTotal": 0,
+ "cellsWithSteers": 0,
+ "cellsWithUnavailableSteers": 9,
+ "wavesMean": 3.137,
+ "maxConcurrencyMax": 2,
+ "utilizationMean": 0.474,
+ "idlePctMean": 52.618,
+ "workersSpawnedTotal": 161,
+ "acceptedTotal": 35,
+ "usdTotal": 3.258497,
+ "resolvedCount": 17,
+ "perCell": [
+  {
+   "instanceId": "astropy__astropy-13033",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 2,
+   "utilization": 0.803,
+   "idlePct": 19.7,
+   "resolved": false,
+   "usd": 0.049903
+  },
+  {
+   "instanceId": "django__django-11532",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 2,
+   "utilization": 0.702,
+   "idlePct": 29.8,
+   "resolved": false,
+   "usd": 0.036822
+  },
+  {
+   "instanceId": "matplotlib__matplotlib-20826",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 10,
+   "utilization": 0.801,
+   "idlePct": 19.9,
+   "resolved": false,
+   "usd": 0.224134
+  },
+  {
+   "instanceId": "pydata__xarray-4687",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 5,
+   "utilization": 0.831,
+   "idlePct": 16.9,
+   "resolved": false,
+   "usd": 0.096077
+  },
+  {
+   "instanceId": "pytest-dev__pytest-6197",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 14,
+   "utilization": 0.308,
+   "idlePct": 69.9,
+   "resolved": false,
+   "usd": 0.189169
+  },
+  {
+   "instanceId": "sphinx-doc__sphinx-9658",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 4,
+   "utilization": 0.574,
+   "idlePct": 42.6,
+   "resolved": true,
+   "usd": 0.085226
+  },
+  {
+   "instanceId": "astropy__astropy-13033",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 2,
+   "utilization": 0.833,
+   "idlePct": 16.7,
+   "resolved": false,
+   "usd": 0.033692
+  },
+  {
+   "instanceId": "django__django-11532",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 2,
+   "utilization": 0.546,
+   "idlePct": 45.4,
+   "resolved": false,
+   "usd": 0.079202
+  },
+  {
+   "instanceId": "matplotlib__matplotlib-20826",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 4,
+   "utilization": 0.749,
+   "idlePct": 25.1,
+   "resolved": false,
+   "usd": 0.107538
+  },
+  {
+   "instanceId": "pydata__xarray-4687",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 2,
+   "utilization": 0.722,
+   "idlePct": 27.8,
+   "resolved": true,
+   "usd": 0.040083
+  },
+  {
+   "instanceId": "pytest-dev__pytest-6197",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 4,
+   "utilization": 0.458,
+   "idlePct": 54.2,
+   "resolved": false,
+   "usd": 0.072682
+  },
+  {
+   "instanceId": "sphinx-doc__sphinx-9658",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 5,
+   "utilization": 0.611,
+   "idlePct": 38.9,
+   "resolved": true,
+   "usd": 0.063746
+  },
+  {
+   "instanceId": "astropy__astropy-13033",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 3,
+   "utilization": 0.817,
+   "idlePct": 18.3,
+   "resolved": true,
+   "usd": 0.064968
+  },
+  {
+   "instanceId": "django__django-11532",
+   "arm": "R4",
+   "steers": {
+    "unavailable": "workers/ directory absent under /tmp/claude-1000/-home-drew-code-supervisor-lab/f06fd156-042a-4ef9-bd88-f2ec7f52b90c/scratchpad/hh/gen3/arm-runs/31fc08bd3a/rep-0/runs/django__django-11532/R4/ws/.loops/supervisor/sup-1-4b0be0"
+   },
+   "waves": 0,
+   "utilization": 0,
+   "idlePct": 100,
+   "resolved": false,
+   "usd": 0
+  },
+  {
+   "instanceId": "matplotlib__matplotlib-20826",
+   "arm": "R4",
+   "steers": {
+    "unavailable": "workers/ directory absent under /tmp/claude-1000/-home-drew-code-supervisor-lab/f06fd156-042a-4ef9-bd88-f2ec7f52b90c/scratchpad/hh/gen3/arm-runs/31fc08bd3a/rep-0/runs/matplotlib__matplotlib-20826/R4/ws/.loops/supervisor/sup-1-2c15b1"
+   },
+   "waves": 0,
+   "utilization": 0,
+   "idlePct": 100,
+   "resolved": false,
+   "usd": 0
+  },
+  {
+   "instanceId": "pydata__xarray-4687",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 2,
+   "utilization": 0.473,
+   "idlePct": 52.7,
+   "resolved": true,
+   "usd": 0.043101
+  },
+  {
+   "instanceId": "pytest-dev__pytest-6197",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 13,
+   "utilization": 0.352,
+   "idlePct": 64.8,
+   "resolved": false,
+   "usd": 0.222542
+  },
+  {
+   "instanceId": "sphinx-doc__sphinx-9658",
+   "arm": "R4",
+   "steers": {
+    "unavailable": "workers/ directory absent under /tmp/claude-1000/-home-drew-code-supervisor-lab/f06fd156-042a-4ef9-bd88-f2ec7f52b90c/scratchpad/hh/gen3/arm-runs/31fc08bd3a/rep-0/runs/sphinx-doc__sphinx-9658/R4/ws/.loops/supervisor/sup-1-cd137b"
+   },
+   "waves": 0,
+   "utilization": 0,
+   "idlePct": 100,
+   "resolved": false,
+   "usd": 0
+  },
+  {
+   "instanceId": "astropy__astropy-13033",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 1,
+   "utilization": 0.321,
+   "idlePct": 67.9,
+   "resolved": false,
+   "usd": 0.014523
+  },
+  {
+   "instanceId": "django__django-11532",
+   "arm": "R4",
+   "steers": {
+    "unavailable": "workers/ directory absent under /tmp/claude-1000/-home-drew-code-supervisor-lab/f06fd156-042a-4ef9-bd88-f2ec7f52b90c/scratchpad/hh/gen3/arm-runs/31fc08bd3a/rep-1/runs/django__django-11532/R4/ws/.loops/supervisor/sup-1-c84e92"
+   },
+   "waves": 0,
+   "utilization": 0,
+   "idlePct": 100,
+   "resolved": false,
+   "usd": 0
+  },
+  {
+   "instanceId": "matplotlib__matplotlib-20826",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 2,
+   "utilization": 0.305,
+   "idlePct": 69.5,
+   "resolved": true,
+   "usd": 0.028989
+  },
+  {
+   "instanceId": "pydata__xarray-4687",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 4,
+   "utilization": 0.648,
+   "idlePct": 35.2,
+   "resolved": true,
+   "usd": 0.081441
+  },
+  {
+   "instanceId": "pytest-dev__pytest-6197",
+   "arm": "R4",
+   "steers": {
+    "unavailable": "workers/ directory absent under /tmp/claude-1000/-home-drew-code-supervisor-lab/f06fd156-042a-4ef9-bd88-f2ec7f52b90c/scratchpad/hh/gen3/arm-runs/31fc08bd3a/rep-1/runs/pytest-dev__pytest-6197/R4/ws/.loops/supervisor/sup-1-d7b670"
+   },
+   "waves": 0,
+   "utilization": 0,
+   "idlePct": 100,
+   "resolved": false,
+   "usd": 0
+  },
+  {
+   "instanceId": "sphinx-doc__sphinx-9658",
+   "arm": "R4",
+   "steers": {
+    "unavailable": "workers/ directory absent under /tmp/claude-1000/-home-drew-code-supervisor-lab/f06fd156-042a-4ef9-bd88-f2ec7f52b90c/scratchpad/hh/gen3/arm-runs/31fc08bd3a/rep-1/runs/sphinx-doc__sphinx-9658/R4/ws/.loops/supervisor/sup-1-37e35d"
+   },
+   "waves": 0,
+   "utilization": 0,
+   "idlePct": 100,
+   "resolved": false,
+   "usd": 0
+  },
+  {
+   "instanceId": "astropy__astropy-13033",
+   "arm": "R4",
+   "steers": {
+    "unavailable": "workers/ directory absent under /tmp/claude-1000/-home-drew-code-supervisor-lab/f06fd156-042a-4ef9-bd88-f2ec7f52b90c/scratchpad/hh/gen3/arm-runs/a7a2a982e5/rep-0/runs/astropy__astropy-13033/R4/ws/.loops/supervisor/sup-1-4b8aff"
+   },
+   "waves": 0,
+   "utilization": 0,
+   "idlePct": 100,
+   "resolved": false,
+   "usd": 0
+  },
+  {
+   "instanceId": "django__django-11532",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 2,
+   "utilization": 0.447,
+   "idlePct": 55.3,
+   "resolved": true,
+   "usd": 0.042501
+  },
+  {
+   "instanceId": "matplotlib__matplotlib-20826",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 2,
+   "utilization": 0.233,
+   "idlePct": 76.7,
+   "resolved": false,
+   "usd": 0.028501
+  },
+  {
+   "instanceId": "pydata__xarray-4687",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 3,
+   "utilization": 0.847,
+   "idlePct": 15.3,
+   "resolved": true,
+   "usd": 0.070831
+  },
+  {
+   "instanceId": "pytest-dev__pytest-6197",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 2,
+   "utilization": 0.14,
+   "idlePct": 86,
+   "resolved": false,
+   "usd": 0.029033
+  },
+  {
+   "instanceId": "sphinx-doc__sphinx-9658",
+   "arm": "R4",
+   "steers": {
+    "unavailable": "workers/ directory absent under /tmp/claude-1000/-home-drew-code-supervisor-lab/f06fd156-042a-4ef9-bd88-f2ec7f52b90c/scratchpad/hh/gen3/arm-runs/a7a2a982e5/rep-0/runs/sphinx-doc__sphinx-9658/R4/ws/.loops/supervisor/sup-1-d97029"
+   },
+   "waves": 0,
+   "utilization": 0,
+   "idlePct": 100,
+   "resolved": false,
+   "usd": 0
+  },
+  {
+   "instanceId": "astropy__astropy-13033",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 2,
+   "utilization": 0.482,
+   "idlePct": 51.8,
+   "resolved": false,
+   "usd": 0.037866
+  },
+  {
+   "instanceId": "django__django-11532",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 3,
+   "utilization": 0.388,
+   "idlePct": 61.2,
+   "resolved": true,
+   "usd": 0.059868
+  },
+  {
+   "instanceId": "matplotlib__matplotlib-20826",
+   "arm": "R4",
+   "steers": {
+    "unavailable": "workers/ directory absent under /tmp/claude-1000/-home-drew-code-supervisor-lab/f06fd156-042a-4ef9-bd88-f2ec7f52b90c/scratchpad/hh/gen3/arm-runs/a7a2a982e5/rep-1/runs/matplotlib__matplotlib-20826/R4/ws/.loops/supervisor/sup-1-eab778"
+   },
+   "waves": 0,
+   "utilization": 0,
+   "idlePct": 100,
+   "resolved": false,
+   "usd": 0
+  },
+  {
+   "instanceId": "pydata__xarray-4687",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 2,
+   "utilization": 0.714,
+   "idlePct": 28.6,
+   "resolved": true,
+   "usd": 0.044491
+  },
+  {
+   "instanceId": "pytest-dev__pytest-6197",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 5,
+   "utilization": 0.566,
+   "idlePct": 43.4,
+   "resolved": false,
+   "usd": 0.108311
+  },
+  {
+   "instanceId": "sphinx-doc__sphinx-9658",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 4,
+   "utilization": 0.359,
+   "idlePct": 64.1,
+   "resolved": true,
+   "usd": 0.060977
+  },
+  {
+   "instanceId": "astropy__astropy-13033",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 2,
+   "utilization": 0.834,
+   "idlePct": 16.6,
+   "resolved": false,
+   "usd": 0.04263
+  },
+  {
+   "instanceId": "django__django-11532",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 3,
+   "utilization": 0.394,
+   "idlePct": 60.6,
+   "resolved": false,
+   "usd": 0.068403
+  },
+  {
+   "instanceId": "matplotlib__matplotlib-20826",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 5,
+   "utilization": 0.588,
+   "idlePct": 41.2,
+   "resolved": false,
+   "usd": 0.110963
+  },
+  {
+   "instanceId": "pydata__xarray-4687",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 3,
+   "utilization": 0.82,
+   "idlePct": 18,
+   "resolved": true,
+   "usd": 0.084503
+  },
+  {
+   "instanceId": "pytest-dev__pytest-6197",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 3,
+   "utilization": 0.281,
+   "idlePct": 71.9,
+   "resolved": false,
+   "usd": 0.079547
+  },
+  {
+   "instanceId": "sphinx-doc__sphinx-9658",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 5,
+   "utilization": 0.649,
+   "idlePct": 35.1,
+   "resolved": true,
+   "usd": 0.068044
+  },
+  {
+   "instanceId": "astropy__astropy-13033",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 4,
+   "utilization": 0.85,
+   "idlePct": 15,
+   "resolved": false,
+   "usd": 0.096642
+  },
+  {
+   "instanceId": "django__django-11532",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 2,
+   "utilization": 0.244,
+   "idlePct": 75.6,
+   "resolved": false,
+   "usd": 0.047655
+  },
+  {
+   "instanceId": "matplotlib__matplotlib-20826",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 7,
+   "utilization": 0.76,
+   "idlePct": 24,
+   "resolved": false,
+   "usd": 0.161885
+  },
+  {
+   "instanceId": "pydata__xarray-4687",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 5,
+   "utilization": 0.528,
+   "idlePct": 47.2,
+   "resolved": true,
+   "usd": 0.084818
+  },
+  {
+   "instanceId": "pytest-dev__pytest-6197",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 5,
+   "utilization": 0.263,
+   "idlePct": 73.7,
+   "resolved": true,
+   "usd": 0.122129
+  },
+  {
+   "instanceId": "sphinx-doc__sphinx-9658",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 2,
+   "utilization": 0.514,
+   "idlePct": 48.6,
+   "resolved": true,
+   "usd": 0.0709
+  },
+  {
+   "instanceId": "astropy__astropy-13033",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 3,
+   "utilization": 0.789,
+   "idlePct": 21.1,
+   "resolved": {
+    "unavailable": "judge.json absent"
+   },
+   "usd": 0.079491
+  },
+  {
+   "instanceId": "astropy__astropy-13033",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 2,
+   "utilization": 0.846,
+   "idlePct": 15.4,
+   "resolved": {
+    "unavailable": "judge.json absent"
+   },
+   "usd": 0.040525
+  },
+  {
+   "instanceId": "astropy__astropy-13033",
+   "arm": "R4",
+   "steers": 0,
+   "waves": 3,
+   "utilization": 0.782,
+   "idlePct": 21.8,
+   "resolved": {
+    "unavailable": "judge.json absent"
+   },
+   "usd": 0.084145
+  }
+ ]
+}