toolbench 0.2.1__tar.gz → 0.4.0__tar.gz

toolbench-0.4.0/CHANGELOG.md

@@ -0,0 +1,93 @@
+# Changelog
+
+All notable changes to `toolbench` will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/), and this project adheres to [Semantic Versioning](https://semver.org/).
+
+---
+
+## [Unreleased]
+
+## [0.4.0] — 2026-07-28
+
+### Added
+
+- **`toolbench export`** — turn a completed run into something you can share.
+  A run directory is a working artifact: gigabytes of transcripts and
+  intermediate data carrying absolute paths from the machine that produced it,
+  which is not a thing you can attach to a paper or render a results page from.
+  `export` writes two layers instead:
+
+  - `trials.jsonl` — one flat, denormalized, **schema-versioned** row per trial
+    (cell coordinates, score, pass/fail against the rubric's own threshold,
+    per-stage credits/weights/metrics, telemetry, provenance). Kilobytes.
+    Denormalized on purpose so a consumer needs no join logic and no knowledge
+    of toolbench's internal layout.
+  - `bundle/` — the graded evidence behind those rows: per-trial answer files,
+    audit logs, run summaries, manifest. Megabytes.
+
+  Transcripts are excluded by default (`--include-transcripts` opts in): they
+  dominate the size and, being binary, are copied verbatim rather than scrubbed.
+  Machine-specific absolute paths are rewritten to `${HOME}` / `${RUN}`
+  placeholders unless `--no-scrub` is passed. `--archive` also writes a
+  `.tar.gz`. On a representative 5-trial run this is 2.7 GB -> 13 MB (1.8 MB
+  compressed).
+
+  `schema_version` is the compatibility contract: additive changes bump the
+  minor, anything that moves or retypes an existing field bumps the major.
+
+## [0.3.0] — 2026-07-24
+
+### Added
+
+- **MCP preflight verification.** Before any trial runs (and in `--dry-run`), a
+  run now verifies that every MCP-serving harness × tools-loadout can actually
+  reach its tools: it starts the loadout's `toolbase serve --profile …` exactly
+  as the CLI runtime would and completes an MCP `initialize` + `tools/list`
+  handshake. If the profile serves zero tools, the handshake times out, or the
+  resolved tools are missing, the run **aborts with exit 2 before executing a
+  single trial** and prints which cell failed. Previously a `tools` loadout that
+  could not reach its tools ran the entire arm silently tool-less and still
+  graded as a valid tools result. Verification is gated to the runtimes that
+  serve toolbase over MCP (`claude_code`, `codex`) via
+  `runtime_serves_toolbase_mcp`; in-process runtimes (orchestral) resolve tools
+  directly and are unaffected. New helpers: `verify_toolbase_mcp`,
+  `runtime_serves_toolbase_mcp`.
+
+### Fixed
+
+- **`_toolbase_command()` now fails loudly.** When the `toolbase` executable
+  cannot be located (neither on `PATH` nor beside the running Python), it raises
+  with an actionable message instead of returning a bare `"toolbase"`. The bare
+  fallback silently produced a broken `.mcp.json` whose stdio server failed to
+  launch (`command not found` on the child's `PATH`), leaving the agent with
+  "No such tool available" while the trial still graded as if tools were
+  present — a failure mode that surfaced when an env's `toolbase` install was
+  in flux. Combined with the preflight above, an unresolvable toolbase is now
+  caught before the benchmark begins.
+
+## [0.2.1] — 2026-07-24
+
+Runtime and metrics hardening surfaced by the first publication benchmarking campaign.
+
+### Added
+
+- **`SESSION_LIMIT` failure mode.** A subscription coding-agent CLI (`claude-code` / `codex`) that refuses a request because the logged-in account hit its plan's session/usage quota is no longer misfiled as an `AGENT_CRASH` scored 0. It is classified as a distinct `SESSION_LIMIT` mode (detected from the CLI limit text, checked before `RATE_LIMITED` so the API's `insufficient_quota` still maps to `RATE_LIMITED`), dropped from the scored population (reach / pass@k / stage funnel / paired deltas) via `EXCLUDED_FROM_METRICS`, and re-run on `resume`. The runner aborts the remaining queue on the first one — every later trial would fail identically until the quota resets — and reports the abort reason. Excluded rows stay on disk and surface as per-cell and run-level excluded counts. ([#23](https://github.com/tonymenzo/toolbench/pull/23))
+- `gpt-5.6-sol` subscription API-equivalent pricing entry for cost estimation.
+
+### Changed
+
+- **Benchmark agents can no longer read Toolbase toolkit sources off disk.** A trial's `protected_paths` now also deny the agent read access to editable toolkit checkouts (`source_path` in each cached toolkit's `.install_meta.yaml`) and toolkit runtime data directories (`base_directory` in toolkit configs). Without this, an agent in a `core_only` control arm could bypass the MCP interface and import the real tool implementations straight off the filesystem, contaminating the control. MCP server children are not confined by the agent's shell filesystem profile, so declared Toolbase loadouts still execute normally.
+
+### Fixed
+
+- **`claude-code` runtime ignored the `--models` sweep** — every cell ran the harness provider's default model instead of the requested one, so a model axis silently collapsed to a single model. The runtime now honors the swept model.
+- **Toolbase MCP command resolution.** Child MCP processes launched a bare `toolbase` command, which failed when Toolbench was started with an absolute virtualenv Python whose `bin/` was not on `PATH`. The command is now resolved to an absolute path (`PATH` → the executable beside the running Python → bare command as a last-resort fallback so the CLI can report a normal MCP startup error).
+
+## [0.2.0] — 2026-06-05
+
+- Continuous / non-gating scoring, `codex` and `claude-code` subscription runtimes, selectable LLM judge, trajectory integrity auditor, subscription cost estimation, and parallel-run hardening.
+
+---
+
+Versions before 0.2.0 predate this changelog; see the git history and release commit subjects for their contents.

{toolbench-0.2.1 → toolbench-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: toolbench
-Version: 0.2.1
+Version: 0.4.0
 Summary: A platform and CLI for building benchmarks for agentic tools and harnesses.
 Project-URL: Homepage, https://github.com/tonymenzo/toolbench
 Project-URL: Repository, https://github.com/tonymenzo/toolbench

{toolbench-0.2.1 → toolbench-0.4.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "toolbench"
-version = "0.2.1"
+version = "0.4.0"
 description = "A platform and CLI for building benchmarks for agentic tools and harnesses."
 readme = "README.md"
 requires-python = ">=3.12"

{toolbench-0.2.1 → toolbench-0.4.0}/tests/test_codex_runtime.py

@@ -4,7 +4,7 @@ from pathlib import Path
 
 from toolbench.core.harness import Harness
 from toolbench.core.llm_factory import SubscriptionLLM
-from toolbench.core.runtime import CodexAgent, _codex_factory
+from toolbench.core.runtime import CodexAgent, _codex_factory, _toolbase_command
 
 
 def _agent(tmp_path: Path, **kw) -> CodexAgent:
@@ -29,6 +29,12 @@ def test_toolbase_mcp_is_auto_approved_for_noninteractive_exec(tmp_path):
     assert any("default_tools_approval_mode=\"approve\"" in x for x in args)
 
 
+def test_toolbase_mcp_command_resolves_to_an_executable():
+    command = Path(_toolbase_command())
+    assert command.is_absolute()
+    assert command.is_file()
+
+
 def test_resume_uses_only_resume_supported_flags(tmp_path):
     a = _agent(tmp_path)
     a.thread_id = "thread-123"

{toolbench-0.2.1 → toolbench-0.4.0}/tests/test_crash_classifier.py

@@ -9,6 +9,7 @@ sys.path.insert(0, str(REPO_ROOT))
 from toolbench.core.crash_classifier import classify_crash  # noqa: E402
 from toolbench.core.failure_modes import (  # noqa: E402
     AGENT_CRASH, CONTEXT_LENGTH_EXCEEDED, MODEL_FORMAT_CRASH, RATE_LIMITED,
+    SESSION_LIMIT,
 )
 
 
@@ -155,6 +156,45 @@ class TestRateLimitCrash(unittest.TestCase):
         self.assertEqual(mode, MODEL_FORMAT_CRASH)
 
 
+class TestSessionLimitCrash(unittest.TestCase):
+    """A subscription coding-agent CLI hitting the account's plan quota is a
+    SESSION_LIMIT — distinct from a 429 throttle and never a capability
+    failure. Checked before RATE_LIMITED so it can't be misfiled as one."""
+
+    def test_claude_code_session_limit_message(self):
+        # Verbatim shape of the claude-code CLI's is_error limit response.
+        exc = RuntimeError(
+            "claude_code runtime: claude reported is_error (success): "
+            "You've hit your session limit · resets 4:20am (America/Chicago)")
+        mode, reason = classify_crash(exc, "")
+        self.assertEqual(mode, SESSION_LIMIT)
+        self.assertIn("limit", reason.lower())
+
+    def test_usage_limit_message(self):
+        exc = RuntimeError("You've reached your usage limit for this plan")
+        mode, _ = classify_crash(exc, "")
+        self.assertEqual(mode, SESSION_LIMIT)
+
+    def test_session_limit_wins_over_rate_limit(self):
+        # A message mentioning both must classify as SESSION_LIMIT (checked
+        # first), not RATE_LIMITED — a quota reset is not a transient throttle.
+        exc = RuntimeError("session limit reached (rate limit 429)")
+        mode, _ = classify_crash(exc, "")
+        self.assertEqual(mode, SESSION_LIMIT)
+
+    def test_api_insufficient_quota_stays_rate_limited(self):
+        # The API's billing-quota error must NOT be swept into SESSION_LIMIT;
+        # it remains a RATE_LIMITED (retried/backed-off) outcome.
+        exc = RuntimeError("You exceeded your current quota: insufficient_quota")
+        mode, _ = classify_crash(exc, "")
+        self.assertEqual(mode, RATE_LIMITED)
+
+    def test_unrelated_crash_is_not_session_limit(self):
+        exc = RuntimeError("ValueError: could not parse geometry.yaml")
+        mode, _ = classify_crash(exc, "")
+        self.assertEqual(mode, AGENT_CRASH)
+
+
 class TestProviderRejectedToolCall(unittest.TestCase):
     """Groq validates tool-call arguments server-side.
 

toolbench-0.4.0/tests/test_export.py

@@ -0,0 +1,100 @@
+"""Export contract tests.
+
+The export is what leaves the machine, so the two properties worth pinning are
+(1) the flat schema stays stable and self-describing, and (2) scrubbing actually
+removes machine-specific paths — a silent regression there publishes someone's
+home directory.
+"""
+import gzip
+import json
+from pathlib import Path
+
+import pytest
+
+from toolbench.core.export import SCHEMA_VERSION, export_run
+
+
+def _make_run(tmp_path: Path, home_marker: str) -> Path:
+    run = tmp_path / "runs" / "2026-01-01T00-00-00_demo_stub_run"
+    (run / "trials" / "core_only__n000__seed1001" / "artifacts" / "results").mkdir(
+        parents=True)
+    trial_dir = run / "trials" / "core_only__n000__seed1001"
+
+    (run / "trials.jsonl").write_text(json.dumps({
+        "trial_id": "core_only__n000__seed1001",
+        "model": "stub", "harness": "h/default", "loadout": "core_only",
+        "variant": "v", "seed": 1001, "ok": True, "score": 0.95,
+        "stages": {"a": True}, "stage_credits": {"a": 1.0},
+        "wall_clock_s": 12.5, "cost_usd": 0.25, "output_tokens": 100,
+        "tool_calls": 3, "tool_errors": 0, "failure_mode": None,
+    }) + "\n")
+    (run / "manifest.json").write_text(json.dumps({
+        "run_id": run.name, "task": "demo", "versions": {"toolbench": "0.3.0"},
+    }))
+    (run / "summary.json").write_text(json.dumps({
+        "reach_weights": {"pass_threshold": 0.9},
+    }))
+    (run / "summary.txt").write_text(f"summary referencing {home_marker}/secret/path\n")
+    (trial_dir / "trial.json").write_text(json.dumps({
+        "grade": {"stage_grades": [
+            {"id": "a", "weight": 1.0, "credit": 1.0, "evidence": "ok",
+             "metrics": {"check_a": {"closeness": 0.5, "distance": 1.25,
+                                     "per_item": {"x": 1}}}},
+        ]}
+    }))
+    (trial_dir / "audit.txt").write_text(f"ran in {home_marker}/work/sandbox\n")
+    (trial_dir / "artifacts" / "results" / "answer.json").write_text('{"v": 1}')
+    with gzip.open(trial_dir / "transcript.jsonl.gz", "wt") as fh:
+        fh.write('{"big": "transcript"}\n')
+    return run
+
+
+def test_flat_schema_and_scrub(tmp_path):
+    home = str(Path.home())
+    run = _make_run(tmp_path, home)
+    out = tmp_path / "export"
+    rep = export_run(run, out, scrub=True)
+
+    assert rep["n_trials"] == 1
+    rows = [json.loads(l) for l in (out / "trials.jsonl").read_text().splitlines()]
+    r = rows[0]
+
+    # (1) schema is self-describing and carries the cell coordinates flat
+    assert r["schema_version"] == SCHEMA_VERSION
+    for key in ("run_id", "trial_id", "harness", "loadout", "variant", "model",
+                "seed", "score", "pass_threshold", "passed", "stage_credits",
+                "stage_weights", "stage_metrics", "tokens", "provenance"):
+        assert key in r, f"missing {key}"
+
+    # pass/fail is resolved from the rubric threshold, not hardcoded
+    assert r["pass_threshold"] == 0.9
+    assert r["passed"] is True
+
+    # metrics are flattened out of their per-check nesting; bulk dicts dropped
+    assert r["stage_metrics"]["a"]["closeness"] == 0.5
+    assert "per_item" not in r["stage_metrics"]["a"]
+
+    # (2) no home path survives anywhere in the text output
+    for p in list(out.rglob("*")):
+        if p.is_file() and p.suffix != ".gz":
+            assert home not in p.read_text(), f"unscrubbed home path in {p}"
+
+    # transcripts excluded by default
+    assert not list((out / "bundle").rglob("transcript.jsonl.gz"))
+
+
+def test_transcripts_opt_in_and_no_scrub_flag(tmp_path):
+    home = str(Path.home())
+    run = _make_run(tmp_path, home)
+    out = tmp_path / "export2"
+    export_run(run, out, scrub=False, include_transcripts=True)
+    assert list((out / "bundle").rglob("transcript.jsonl.gz"))
+    # with scrubbing off the raw path is preserved (so the flag is meaningful)
+    assert home in (out / "bundle" / "summary.txt").read_text()
+
+
+def test_rejects_incomplete_run(tmp_path):
+    empty = tmp_path / "runs" / "not_a_run"
+    empty.mkdir(parents=True)
+    with pytest.raises(FileNotFoundError):
+        export_run(empty, tmp_path / "out")

{toolbench-0.2.1 → toolbench-0.4.0}/tests/test_metrics.py

@@ -242,6 +242,18 @@ class TestCostUsd(unittest.TestCase):
         self.assertAlmostEqual(estimate["usd"], 56.0)
         self.assertTrue(estimate["long_context_pricing_applied"])
 
+    def test_gpt56_sol_subscription_api_equivalent(self):
+        estimate = subscription_api_equivalent_cost(
+            "gpt-5.6-sol",
+            input_tokens=1_000_000,
+            cache_read_tokens=1_000_000,
+            output_tokens=1_000_000,
+            initial_input_tokens=272_001,
+        )
+        self.assertAlmostEqual(estimate["usd"], 56.0)
+        self.assertTrue(estimate["long_context_pricing_applied"])
+        self.assertIn("gpt-5.6-sol", estimate["source"])
+
     def test_unknown_subscription_model_is_not_guessed(self):
         self.assertIsNone(subscription_api_equivalent_cost("future-model"))
 

{toolbench-0.2.1 → toolbench-0.4.0}/tests/test_paired_deltas.py

@@ -78,5 +78,45 @@ class TestPairedDeltas(unittest.TestCase):
         self.assertAlmostEqual(summary["paired_deltas"][0]["reach_delta"], 0.45)
 
 
+class TestExcludedFromMetrics(unittest.TestCase):
+    """SESSION_LIMIT trials are dropped from the scored population (n / reach /
+    pass / paired deltas) but surfaced as an explicit excluded count, so a
+    subscription-quota termination never pollutes the metrics as a score-0."""
+
+    def _rows(self):
+        from toolbench.core.failure_modes import SESSION_LIMIT
+        rows = []
+        # cell (m, A): 2 genuine passes (reach 1.0) + 2 quota terminations.
+        for seed in (1, 2):
+            rows.append(_row("m", "A", seed, 1, 1))
+        for seed in (3, 4):
+            rows.append(_row("m", "A", seed, 0, 0,
+                             failure_mode=SESSION_LIMIT, stages={}, score=0.0))
+        # cell (m, B): 2 genuine trials sharing seeds 1,2 with A.
+        for seed in (1, 2):
+            rows.append(_row("m", "B", seed, 1, 0))
+        return rows
+
+    def test_excluded_from_scored_population(self):
+        summary = aggregate(self._rows(), k=4, stage_order=STAGE_ORDER,
+                            stage_weights=WEIGHTS)
+        self.assertEqual(summary["n_total_trials"], 4)      # 2 A + 2 B scored
+        self.assertEqual(summary["n_excluded_trials"], 2)   # 2 quota rows
+        cell_a = next(c for c in summary["cells"] if c["condition"] == "A")
+        self.assertEqual(cell_a["n"], 2)
+        self.assertEqual(cell_a["n_excluded"], 2)
+        # Reach reflects ONLY the 2 genuine passes, not diluted to 0.5 by the
+        # 2 excluded score-0 quota rows.
+        self.assertAlmostEqual(cell_a["reach_bar_k"], 1.0)
+
+    def test_excluded_not_in_paired_deltas(self):
+        # The A−B delta pairs only scored seeds (1, 2); A's excluded quota
+        # rows on seeds 3, 4 must not create phantom pairs or shift the delta.
+        summary = aggregate(self._rows(), k=4, stage_order=STAGE_ORDER,
+                            stage_weights=WEIGHTS)
+        self.assertEqual(len(summary["paired_deltas"]), 1)
+        self.assertEqual(summary["paired_deltas"][0]["n_pairs"], 2)
+
+
 if __name__ == "__main__":
     unittest.main()

{toolbench-0.2.1 → toolbench-0.4.0}/tests/test_resume.py

@@ -28,6 +28,23 @@ def test_partition_splits_resolution_errors():
     assert [r["trial_id"] for r in retryable] == ["b"]
 
 
+def test_partition_retries_session_limit():
+    # A subscription session/usage-quota termination took no real
+    # measurement, so a resume must re-run it (like resolution_error) rather
+    # than freezing its score-0 record. A real AGENT_CRASH stays kept.
+    from toolbench.core.failure_modes import SESSION_LIMIT
+    rows = [
+        {"trial_id": "a", "failure_mode": "NONE"},
+        {"trial_id": "b", "failure_mode": SESSION_LIMIT},
+        {"trial_id": "c", "failure_mode": "resolution_error"},
+        {"trial_id": "d", "failure_mode": "AGENT_CRASH"},
+        {"trial_id": "e", "failure_mode": "MODEL_STOPPED_EARLY"},
+    ]
+    kept, retryable = _partition_resume_rows(rows)
+    assert [r["trial_id"] for r in kept] == ["a", "d", "e"]
+    assert sorted(r["trial_id"] for r in retryable) == ["b", "c"]
+
+
 def test_budget_precharge_counts_toward_cap():
     b = Budget(10.0)
     b.precharge(9.5)

toolbench-0.4.0/tests/test_runner_toolbase_protection.py

@@ -0,0 +1,29 @@
+from pathlib import Path
+
+from toolbench.core.runner import _toolbase_protected_paths
+
+
+def test_toolbase_and_editable_sources_are_protected(tmp_path, monkeypatch):
+    toolbase_home = tmp_path / ".toolbase"
+    slot = toolbase_home / "cache" / "physics-kit" / "editable"
+    slot.mkdir(parents=True)
+    source = tmp_path / "physics-kit-source"
+    source.mkdir()
+    runtime_data = tmp_path / "physics-kit-runtime"
+    runtime_data.mkdir()
+    (slot / ".install_meta.yaml").write_text(
+        "editable: true\n"
+        f"source_path: {source}\n"
+    )
+    monkeypatch.setenv("TOOLBASE_HOME", str(toolbase_home))
+    config_dir = toolbase_home / "config"
+    config_dir.mkdir()
+    (config_dir / "physics-kit.yaml").write_text(
+        f"base_directory: {runtime_data}\n"
+    )
+
+    paths = _toolbase_protected_paths()
+
+    assert str(toolbase_home.resolve()) in paths
+    assert str(source.resolve()) in paths
+    assert str(runtime_data.resolve()) in paths

{toolbench-0.2.1 → toolbench-0.4.0}/tests/test_trial_loop.py

@@ -49,10 +49,13 @@ class _FakeRunner:
     """Stands in for TrialRunner: records execution, optionally charges
     the budget per trial so abort behavior can be exercised."""
 
-    def __init__(self, cost_per_trial=0.0, barrier=None):
+    def __init__(self, cost_per_trial=0.0, barrier=None, fail_mode=None):
         self.cost = cost_per_trial
         self.ran = []
         self.barrier = barrier
+        # When set, every trial returns this failure_mode with score 0 (used to
+        # exercise the session-limit abort path).
+        self.fail_mode = fail_mode
         self._lock = threading.Lock()
 
     def run_trial(self, *, model_cfg, benchmark, harness, loadout, variant,
@@ -75,6 +78,14 @@ class _FakeRunner:
             aborted = True
         from toolbench.core.task import Grade
         from toolbench.core.trajectory import Trajectory
+        if self.fail_mode is not None:
+            grade = Grade(score=0.0, stages={}, stage_grades=[],
+                          failure_mode=self.fail_mode, judge_kind="rule")
+            return SimpleNamespace(
+                trial_id=trial_id, ok=False, score=0.0, grade=grade,
+                trajectory=Trajectory(), wall_clock_s=0.0, cost_usd=self.cost,
+                aborted_by_budget=aborted, error=None, attempts=1, nudges=0,
+                rate_limit_retries=0, transient_retries=0)
         grade = Grade(score=1.0, stages={"s0": True}, stage_grades=[],
                       failure_mode="NONE", judge_kind="rule")
         return SimpleNamespace(
@@ -98,17 +109,19 @@ class TestRunTrialLoop(unittest.TestCase):
 
     def test_serial_runs_everything(self):
         runner = _FakeRunner()
-        records, aborted = self._loop(runner, Budget(None))
+        records, aborted, reason = self._loop(runner, Budget(None))
         self.assertEqual(len(records), 6)
         self.assertFalse(aborted)
+        self.assertIsNone(reason)
 
     def test_budget_abort_is_balanced_across_cells(self):
         # $1/trial, $3.5 cap → abort fires on trial 4 (seed-major order:
         # A1 B1 A2 B2 ...), leaving each loadout with the SAME number of
         # completed trials instead of one loadout hogging the budget.
         runner = _FakeRunner(cost_per_trial=1.0)
-        records, aborted = self._loop(runner, Budget(3.5))
+        records, aborted, reason = self._loop(runner, Budget(3.5))
         self.assertTrue(aborted)
+        self.assertEqual(reason, "budget")
         per_cell = {}
         for r in records:
             if not r["aborted_by_budget"]:
@@ -119,22 +132,42 @@ class TestRunTrialLoop(unittest.TestCase):
     def test_parallel_runs_everything_and_is_concurrent(self):
         barrier = threading.Barrier(2)
         runner = _FakeRunner(barrier=barrier)
-        records, aborted = self._loop(runner, Budget(None), parallel=2,
-                                      n_loadouts=2, seeds=(1, 2, 3))
+        records, aborted, reason = self._loop(runner, Budget(None), parallel=2,
+                                              n_loadouts=2, seeds=(1, 2, 3))
         # The barrier requires 2 trials in flight simultaneously; if the
         # loop were serial this would deadlock (and time out the wait).
         self.assertEqual(len(records), 6)
         self.assertFalse(aborted)
+        self.assertIsNone(reason)
         self.assertEqual(len(runner.ran), 6)
 
     def test_parallel_budget_abort_stops_launching(self):
         runner = _FakeRunner(cost_per_trial=1.0)
-        records, aborted = self._loop(runner, Budget(2.5), parallel=2,
-                                      seeds=(1, 2, 3, 4, 5))
+        records, aborted, reason = self._loop(runner, Budget(2.5), parallel=2,
+                                              seeds=(1, 2, 3, 4, 5))
         self.assertTrue(aborted)
+        self.assertEqual(reason, "budget")
         # 10 trials enumerated; the abort must prevent most of the tail.
         self.assertLess(len(runner.ran), 10)
 
+    def test_session_limit_aborts_and_stops_launching(self):
+        # A subscription session/usage-quota termination must halt the queue
+        # (every remaining trial would fail identically until the quota
+        # resets) and report the reason as "session_limit", not "budget".
+        # cost_per_trial slows each trial (as in the budget tests) so the
+        # abort demonstrably beats the queue draining; Budget(None) keeps the
+        # abort attributable to the session limit, not the budget.
+        runner = _FakeRunner(cost_per_trial=1.0, fail_mode="SESSION_LIMIT")
+        records, aborted, reason = self._loop(runner, Budget(None), parallel=2,
+                                              seeds=(1, 2, 3, 4, 5))
+        self.assertTrue(aborted)
+        self.assertEqual(reason, "session_limit")
+        # 10 trials enumerated; the abort must prevent most of the tail from
+        # ever launching (only the already-in-flight one(s) get recorded).
+        self.assertLess(len(runner.ran), 10)
+        self.assertTrue(all(r["failure_mode"] == "SESSION_LIMIT"
+                            for r in records))
+
 
 if __name__ == "__main__":
     unittest.main()

{toolbench-0.2.1 → toolbench-0.4.0}/toolbench/__init__.py

@@ -2,7 +2,7 @@
 
 from pathlib import Path
 
-__version__ = "0.2.1"
+__version__ = "0.4.0"
 
 # Repository root (the directory that contains this `toolbench/` package in an
 # editable install). The CLI reads `REPO_ROOT/.env` for provider keys + tool