toolcallbench 1.0.0__tar.gz

toolcallbench-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 ToolCallBench contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

toolcallbench-1.0.0/PKG-INFO

@@ -0,0 +1,113 @@
+Metadata-Version: 2.4
+Name: toolcallbench
+Version: 1.0.0
+Summary: Benchmark how reliably local LLMs call tools on your own repo and hardware
+Author-email: Herin Patel <herinpatel15113@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/herinpatel15/ToolCallBench
+Project-URL: Issues, https://github.com/herinpatel15/ToolCallBench/issues
+Keywords: llm,benchmark,tool-calling,ollama,function-calling
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pyyaml>=6
+Requires-Dist: jsonschema>=4
+Requires-Dist: pytest>=7
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: build; extra == "dev"
+Requires-Dist: twine; extra == "dev"
+Dynamic: license-file
+
+# ToolCallBench
+
+Find out which local language model calls tools reliably **on your own machine**.
+
+ToolCallBench gives a model six tools (read, list, search, edit, patch, run a command) and 20
+small tasks on a sample Python project. It checks the results with plain code, not with
+another model, and writes a report that compares models and quantization levels.
+
+It is tuned for a **MacBook M4 with 16 GB of memory** running Ollama, and works with any
+OpenAI-compatible local server (llama.cpp, LM Studio, vLLM).
+
+## Install and run
+
+```bash
+pip install toolcallbench
+
+mkdir my-bench && cd my-bench
+toolcallbench init                                  # creates configs/ with example settings
+```
+
+Edit `configs/my-models.yaml` and list the models you have pulled in Ollama, then:
+
+```bash
+toolcallbench doctor --probe --config configs/my-models.yaml   # check Ollama, models, tool calling
+toolcallbench run --config configs/my-models.yaml              # run the 20 tasks
+toolcallbench sessions                                         # list past runs
+toolcallbench report --session <id>                            # Markdown + HTML report in ./results/reports
+```
+
+Every setting can also be given as an argument, and arguments override the config file:
+
+```bash
+toolcallbench run --model qwen3:8b --model gemma4:e4b --repeats 3 --num-ctx 16384 --groups read,edit
+toolcallbench run --help
+```
+
+Other commands: `toolcallbench list-tasks`, `toolcallbench validate-tasks`, `toolcallbench init --with-tasks`
+(copies the tasks and sample repo so you can edit them; a folder with its own `tasks/` and
+`fixtures/repo/` uses them instead of the built-in ones). Results are written to `./results`.
+
+No model yet? `toolcallbench run --backend scripted --repeats 1` runs the whole pipeline without one.
+
+Ollama must be running (`ollama serve`) with the models pulled (`ollama pull qwen3:8b`).
+On a 16 GB Mac, start from the `m4-16gb.yaml` example config. See [docs/USER_GUIDE.md](docs/USER_GUIDE.md).
+
+### Develop
+
+```bash
+git clone <repo> && cd toolcallbench
+python3 -m venv .venv && source .venv/bin/activate
+pip install -e ".[dev]"
+python -m pytest
+```
+
+## What a report tells you
+
+- Pass rate overall and for read, edit, run, chain and failure tasks
+- Why runs failed (13 failure types, such as `invalid_json`, `text_tool_call`, `patch_failed`)
+- Tokens per second, seconds per task, and calls compared with a reference solution
+- Every setting that affects the result (context length, seed, KV cache type, versions)
+
+## Status
+
+Version 1.0.0. The harness, tasks, scoring, backends (checked against mock servers) and
+reports are covered by automated tests. **No real model has been benchmarked yet**; run it on
+your Mac to get the first numbers (`docs/TEST_PLAN.md` lists exactly what was and was not tested).
+
+## Documentation
+
+| Document | Purpose |
+| --- | --- |
+| [BUILD_FROM_SCRATCH.md](BUILD_FROM_SCRATCH.md) | Day-by-day guide to build this project yourself, with all code |
+| [docs/PRD.md](docs/PRD.md) | Why it exists, who it is for, requirements |
+| [docs/TECH_SPEC.md](docs/TECH_SPEC.md) | Exact behavior: tools, scoring, config, storage |
+| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Diagrams: context, components, data flow, sequence, states, storage |
+| [docs/TEST_PLAN.md](docs/TEST_PLAN.md) | What is tested and how requirements map to tests |
+| [docs/SECURITY.md](docs/SECURITY.md) | What protects you and what does not |
+| [docs/DECISIONS.md](docs/DECISIONS.md) | Why it is built this way |
+| [docs/USER_GUIDE.md](docs/USER_GUIDE.md) | Install, run, read the report, troubleshoot |
+| [docs/CLI_REFERENCE.md](docs/CLI_REFERENCE.md) | Every command, option and config key |
+| [docs/MODEL_GUIDE_M4_16GB.md](docs/MODEL_GUIDE_M4_16GB.md) | Memory budget and model choices for 16 GB |
+| [docs/TASK_CATALOG.md](docs/TASK_CATALOG.md) | All 20 tasks (generated) |
+| [docs/ROADMAP.md](docs/ROADMAP.md), [CHANGELOG.md](CHANGELOG.md), [CONTRIBUTING.md](CONTRIBUTING.md) | Project housekeeping |
+
+## License
+
+MIT

toolcallbench-1.0.0/README.md

@@ -0,0 +1,87 @@
+# ToolCallBench
+
+Find out which local language model calls tools reliably **on your own machine**.
+
+ToolCallBench gives a model six tools (read, list, search, edit, patch, run a command) and 20
+small tasks on a sample Python project. It checks the results with plain code, not with
+another model, and writes a report that compares models and quantization levels.
+
+It is tuned for a **MacBook M4 with 16 GB of memory** running Ollama, and works with any
+OpenAI-compatible local server (llama.cpp, LM Studio, vLLM).
+
+## Install and run
+
+```bash
+pip install toolcallbench
+
+mkdir my-bench && cd my-bench
+toolcallbench init                                  # creates configs/ with example settings
+```
+
+Edit `configs/my-models.yaml` and list the models you have pulled in Ollama, then:
+
+```bash
+toolcallbench doctor --probe --config configs/my-models.yaml   # check Ollama, models, tool calling
+toolcallbench run --config configs/my-models.yaml              # run the 20 tasks
+toolcallbench sessions                                         # list past runs
+toolcallbench report --session <id>                            # Markdown + HTML report in ./results/reports
+```
+
+Every setting can also be given as an argument, and arguments override the config file:
+
+```bash
+toolcallbench run --model qwen3:8b --model gemma4:e4b --repeats 3 --num-ctx 16384 --groups read,edit
+toolcallbench run --help
+```
+
+Other commands: `toolcallbench list-tasks`, `toolcallbench validate-tasks`, `toolcallbench init --with-tasks`
+(copies the tasks and sample repo so you can edit them; a folder with its own `tasks/` and
+`fixtures/repo/` uses them instead of the built-in ones). Results are written to `./results`.
+
+No model yet? `toolcallbench run --backend scripted --repeats 1` runs the whole pipeline without one.
+
+Ollama must be running (`ollama serve`) with the models pulled (`ollama pull qwen3:8b`).
+On a 16 GB Mac, start from the `m4-16gb.yaml` example config. See [docs/USER_GUIDE.md](docs/USER_GUIDE.md).
+
+### Develop
+
+```bash
+git clone <repo> && cd toolcallbench
+python3 -m venv .venv && source .venv/bin/activate
+pip install -e ".[dev]"
+python -m pytest
+```
+
+## What a report tells you
+
+- Pass rate overall and for read, edit, run, chain and failure tasks
+- Why runs failed (13 failure types, such as `invalid_json`, `text_tool_call`, `patch_failed`)
+- Tokens per second, seconds per task, and calls compared with a reference solution
+- Every setting that affects the result (context length, seed, KV cache type, versions)
+
+## Status
+
+Version 1.0.0. The harness, tasks, scoring, backends (checked against mock servers) and
+reports are covered by automated tests. **No real model has been benchmarked yet**; run it on
+your Mac to get the first numbers (`docs/TEST_PLAN.md` lists exactly what was and was not tested).
+
+## Documentation
+
+| Document | Purpose |
+| --- | --- |
+| [BUILD_FROM_SCRATCH.md](BUILD_FROM_SCRATCH.md) | Day-by-day guide to build this project yourself, with all code |
+| [docs/PRD.md](docs/PRD.md) | Why it exists, who it is for, requirements |
+| [docs/TECH_SPEC.md](docs/TECH_SPEC.md) | Exact behavior: tools, scoring, config, storage |
+| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Diagrams: context, components, data flow, sequence, states, storage |
+| [docs/TEST_PLAN.md](docs/TEST_PLAN.md) | What is tested and how requirements map to tests |
+| [docs/SECURITY.md](docs/SECURITY.md) | What protects you and what does not |
+| [docs/DECISIONS.md](docs/DECISIONS.md) | Why it is built this way |
+| [docs/USER_GUIDE.md](docs/USER_GUIDE.md) | Install, run, read the report, troubleshoot |
+| [docs/CLI_REFERENCE.md](docs/CLI_REFERENCE.md) | Every command, option and config key |
+| [docs/MODEL_GUIDE_M4_16GB.md](docs/MODEL_GUIDE_M4_16GB.md) | Memory budget and model choices for 16 GB |
+| [docs/TASK_CATALOG.md](docs/TASK_CATALOG.md) | All 20 tasks (generated) |
+| [docs/ROADMAP.md](docs/ROADMAP.md), [CHANGELOG.md](CHANGELOG.md), [CONTRIBUTING.md](CONTRIBUTING.md) | Project housekeeping |
+
+## License
+
+MIT

toolcallbench-1.0.0/pyproject.toml

@@ -0,0 +1,50 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "toolcallbench"
+dynamic = ["version"]
+description = "Benchmark how reliably local LLMs call tools on your own repo and hardware"
+readme = "README.md"
+requires-python = ">=3.10"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "Herin Patel", email = "herinpatel15113@gmail.com" }]
+keywords = ["llm", "benchmark", "tool-calling", "ollama", "function-calling"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Environment :: Console",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3",
+    "Topic :: Software Development :: Testing",
+]
+dependencies = [
+    "pyyaml>=6",
+    "jsonschema>=4",
+    "pytest>=7",  # the benchmark's fixture repository is checked by running its tests
+]
+
+[project.optional-dependencies]
+dev = ["pytest>=7", "build", "twine"]
+
+[project.urls]
+Homepage = "https://github.com/herinpatel15/ToolCallBench"
+Issues = "https://github.com/herinpatel15/ToolCallBench/issues"
+
+[project.scripts]
+toolcallbench = "toolcallbench.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.dynamic]
+version = { attr = "toolcallbench.__version__" }
+
+[tool.setuptools.package-data]
+toolcallbench = ["data/**/*"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+norecursedirs = ["fixtures", "data", ".git", "results"]
+addopts = "-q"

toolcallbench-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

toolcallbench-1.0.0/src/toolcallbench/__init__.py

@@ -0,0 +1,3 @@
+"""ToolCallBench: measure how reliably local LLMs call tools on your own repo and hardware."""
+
+__version__ = "1.0.0"

toolcallbench-1.0.0/src/toolcallbench/__main__.py

@@ -0,0 +1,6 @@
+import sys
+
+from .cli import main
+
+if __name__ == "__main__":
+    sys.exit(main())

toolcallbench-1.0.0/src/toolcallbench/backends/__init__.py

@@ -0,0 +1,43 @@
+"""Model backends and a factory."""
+from __future__ import annotations
+
+from typing import Any
+
+from ..models import Task
+from .base import Backend, BackendError
+from .ollama import OllamaBackend
+from .openai_compat import OpenAICompatBackend
+from .scripted import PROFILES, ScriptedBackend
+
+BACKEND_KINDS = ("ollama", "openai-compat", "scripted")
+
+__all__ = [
+    "BACKEND_KINDS",
+    "PROFILES",
+    "Backend",
+    "BackendError",
+    "OllamaBackend",
+    "OpenAICompatBackend",
+    "ScriptedBackend",
+    "make_backend",
+]
+
+
+def make_backend(
+    kind: str,
+    *,
+    model: str,
+    host: str,
+    options: dict[str, Any],
+    think: bool | None,
+    timeout: float,
+    tasks: list[Task],
+    profile: str = "oracle",
+) -> Backend:
+    if kind == "ollama":
+        return OllamaBackend(model, host=host, options=options, think=think, timeout=timeout)
+    if kind == "openai-compat":
+        return OpenAICompatBackend(model, base_url=host, options=options, timeout=timeout)
+    if kind == "scripted":
+        return ScriptedBackend(tasks, profile=profile)
+    raise ValueError(f"unknown backend '{kind}'; choose from {', '.join(BACKEND_KINDS)}")

toolcallbench-1.0.0/src/toolcallbench/backends/base.py

@@ -0,0 +1,42 @@
+"""Backend interface and a small JSON-over-HTTP helper (standard library only)."""
+from __future__ import annotations
+
+import json
+import urllib.error
+import urllib.request
+from typing import Any
+
+from ..models import BackendResponse
+
+
+class BackendError(RuntimeError):
+    """The model server could not be reached or returned an error. Not a model failure."""
+
+
+class Backend:
+    name = "backend"
+
+    def chat(self, messages: list[dict[str, Any]], tools: list[dict[str, Any]]) -> BackendResponse:
+        raise NotImplementedError
+
+    def check(self) -> None:
+        """Raise BackendError if the server or the model is not usable. Called once before a sweep."""
+
+    def describe(self) -> dict[str, Any]:
+        """Facts about the server and model, recorded in the session settings."""
+        return {"backend": self.name}
+
+
+def request_json(method: str, url: str, payload: dict[str, Any] | None, timeout: float) -> Any:
+    data = json.dumps(payload).encode("utf-8") if payload is not None else None
+    request = urllib.request.Request(url, data=data, method=method, headers={"Content-Type": "application/json"})
+    try:
+        with urllib.request.urlopen(request, timeout=timeout) as response:
+            return json.loads(response.read().decode("utf-8"))
+    except urllib.error.HTTPError as exc:
+        body = exc.read().decode("utf-8", "replace")[:500]
+        raise BackendError(f"HTTP {exc.code} from {url}: {body}") from exc
+    except json.JSONDecodeError as exc:
+        raise BackendError(f"{url} returned invalid JSON") from exc
+    except (urllib.error.URLError, TimeoutError, OSError) as exc:
+        raise BackendError(f"cannot reach {url}: {exc}") from exc

toolcallbench-1.0.0/src/toolcallbench/backends/ollama.py

@@ -0,0 +1,86 @@
+"""Ollama backend: POST /api/chat with tools, non-streaming."""
+from __future__ import annotations
+
+from typing import Any
+
+from ..models import BackendResponse, RawToolCall
+from .base import Backend, BackendError, request_json
+
+
+class OllamaBackend(Backend):
+    name = "ollama"
+
+    def __init__(
+        self,
+        model: str,
+        host: str = "http://localhost:11434",
+        options: dict[str, Any] | None = None,
+        think: bool | None = None,
+        timeout: float = 300.0,
+        keep_alive: str = "30m",
+    ):
+        self.model = model
+        self.host = host.rstrip("/")
+        self.options = dict(options or {})
+        self.think = think
+        self.timeout = timeout
+        self.keep_alive = keep_alive
+
+    # ------------------------------------------------------------ server facts
+
+    def version(self) -> str:
+        return str(request_json("GET", f"{self.host}/api/version", None, self.timeout).get("version", "unknown"))
+
+    def list_models(self) -> list[str]:
+        data = request_json("GET", f"{self.host}/api/tags", None, self.timeout)
+        return [m.get("name", "") for m in data.get("models", [])]
+
+    def show(self) -> dict[str, Any]:
+        return request_json("POST", f"{self.host}/api/show", {"model": self.model}, self.timeout)
+
+    def check(self) -> None:
+        installed = self.list_models()
+        wanted = {self.model, self.model if ":" in self.model else self.model + ":latest"}
+        if not wanted & set(installed):
+            raise BackendError(f"model '{self.model}' is not installed in Ollama; run: ollama pull {self.model}")
+
+    def describe(self) -> dict[str, Any]:
+        info: dict[str, Any] = {"backend": self.name, "host": self.host}
+        try:
+            info["backend_version"] = self.version()
+            shown = self.show()
+            details = shown.get("details", {})
+            info["quant"] = details.get("quantization_level", "")
+            info["parameter_size"] = details.get("parameter_size", "")
+            info["family"] = details.get("family", "")
+            info["capabilities"] = shown.get("capabilities", [])
+        except BackendError:
+            pass
+        return info
+
+    # -------------------------------------------------------------------- chat
+
+    def chat(self, messages: list[dict[str, Any]], tools: list[dict[str, Any]]) -> BackendResponse:
+        payload: dict[str, Any] = {
+            "model": self.model,
+            "messages": messages,
+            "tools": tools,
+            "stream": False,
+            "options": self.options,
+            "keep_alive": self.keep_alive,
+        }
+        if self.think is not None:
+            payload["think"] = self.think
+        data = request_json("POST", f"{self.host}/api/chat", payload, self.timeout)
+        message = data.get("message") or {}
+        calls = []
+        for item in message.get("tool_calls") or []:
+            function = item.get("function") or {}
+            calls.append(RawToolCall(name=str(function.get("name", "")), arguments=function.get("arguments", {})))
+        return BackendResponse(
+            content=message.get("content") or "",
+            tool_calls=calls,
+            eval_tokens=int(data.get("eval_count") or 0),
+            eval_seconds=float(data.get("eval_duration") or 0) / 1e9,
+            prompt_tokens=int(data.get("prompt_eval_count") or 0),
+        )

toolcallbench-1.0.0/src/toolcallbench/backends/openai_compat.py

@@ -0,0 +1,96 @@
+"""OpenAI-compatible backend for llama.cpp `llama-server`, LM Studio, vLLM and similar servers.
+
+Tool arguments arrive as JSON strings here, so this backend exercises the invalid-JSON path.
+"""
+from __future__ import annotations
+
+import json
+import time
+from typing import Any
+
+from ..models import BackendResponse, RawToolCall
+from .base import Backend, BackendError, request_json
+
+
+def to_openai_messages(messages: list[dict[str, Any]]) -> list[dict[str, Any]]:
+    """Convert the runner's Ollama-style messages to the OpenAI chat format (adds tool call ids)."""
+    out: list[dict[str, Any]] = []
+    pending: list[str] = []
+    counter = 0
+    for message in messages:
+        if message["role"] == "assistant" and message.get("tool_calls"):
+            pending = []
+            calls = []
+            for item in message["tool_calls"]:
+                counter += 1
+                call_id = f"call_{counter}"
+                pending.append(call_id)
+                function = item["function"]
+                calls.append(
+                    {
+                        "id": call_id,
+                        "type": "function",
+                        "function": {"name": function["name"], "arguments": json.dumps(function.get("arguments", {}))},
+                    }
+                )
+            out.append({"role": "assistant", "content": message.get("content") or "", "tool_calls": calls})
+        elif message["role"] == "tool":
+            call_id = pending.pop(0) if pending else "call_0"
+            out.append({"role": "tool", "tool_call_id": call_id, "content": message["content"]})
+        else:
+            out.append({"role": message["role"], "content": message.get("content") or ""})
+    return out
+
+
+class OpenAICompatBackend(Backend):
+    name = "openai-compat"
+
+    def __init__(
+        self,
+        model: str,
+        base_url: str = "http://localhost:8080",
+        options: dict[str, Any] | None = None,
+        timeout: float = 300.0,
+    ):
+        self.model = model
+        self.base_url = base_url.rstrip("/")
+        self.options = dict(options or {})
+        self.timeout = timeout
+
+    def check(self) -> None:
+        request_json("GET", f"{self.base_url}/v1/models", None, self.timeout)
+
+    def describe(self) -> dict[str, Any]:
+        return {"backend": self.name, "host": self.base_url}
+
+    def chat(self, messages: list[dict[str, Any]], tools: list[dict[str, Any]]) -> BackendResponse:
+        payload: dict[str, Any] = {
+            "model": self.model,
+            "messages": to_openai_messages(messages),
+            "tools": tools,
+            "stream": False,
+        }
+        for key in ("temperature", "seed"):
+            if key in self.options:
+                payload[key] = self.options[key]
+        started = time.monotonic()
+        data = request_json("POST", f"{self.base_url}/v1/chat/completions", payload, self.timeout)
+        elapsed = time.monotonic() - started
+        try:
+            message = data["choices"][0]["message"]
+        except (KeyError, IndexError, TypeError) as exc:
+            raise BackendError("response has no choices[0].message") from exc
+        calls = []
+        for item in message.get("tool_calls") or []:
+            function = item.get("function") or {}
+            calls.append(RawToolCall(name=str(function.get("name", "")), arguments=function.get("arguments", "")))
+        usage = data.get("usage") or {}
+        timings = data.get("timings") or {}
+        eval_seconds = float(timings["predicted_ms"]) / 1000 if "predicted_ms" in timings else elapsed
+        return BackendResponse(
+            content=message.get("content") or "",
+            tool_calls=calls,
+            eval_tokens=int(usage.get("completion_tokens") or 0),
+            eval_seconds=eval_seconds,
+            prompt_tokens=int(usage.get("prompt_tokens") or 0),
+        )

toolcallbench-1.0.0/src/toolcallbench/backends/scripted.py

@@ -0,0 +1,90 @@
+"""A fake model that replays each task's reference solution, or fails in a chosen way.
+
+It is used to validate the tasks, to test the harness without a real model, and for dry runs.
+"""
+from __future__ import annotations
+
+import json
+from typing import Any
+
+from ..models import BackendResponse, RawToolCall, Task
+from .base import Backend, BackendError
+
+PROFILES = (
+    "oracle",
+    "invalid_json",
+    "unknown_tool",
+    "wrong_tool",
+    "text_call",
+    "no_call",
+    "call_limit",
+    "bad_edit",
+)
+
+
+class ScriptedBackend(Backend):
+    name = "scripted"
+
+    def __init__(self, tasks: list[Task], profile: str = "oracle"):
+        if profile not in PROFILES:
+            raise ValueError(f"unknown profile '{profile}'; choose from {', '.join(PROFILES)}")
+        self.by_prompt = {task.prompt: task for task in tasks}
+        self.profile = profile
+
+    def describe(self) -> dict[str, Any]:
+        return {"backend": self.name, "profile": self.profile}
+
+    def chat(self, messages: list[dict[str, Any]], tools: list[dict[str, Any]]) -> BackendResponse:
+        prompt = next((m["content"] for m in messages if m["role"] == "user"), "")
+        task = self.by_prompt.get(prompt)
+        if task is None:
+            raise BackendError("scripted backend received a prompt that matches no task")
+        turn = sum(1 for m in messages if m["role"] == "assistant")
+        content, calls = self._step(task, turn)
+        return BackendResponse(content=content, tool_calls=calls, eval_tokens=8, eval_seconds=0.01, prompt_tokens=100)
+
+    # ------------------------------------------------------------------ steps
+
+    @staticmethod
+    def _oracle(task: Task, turn: int) -> tuple[str, list[RawToolCall]]:
+        if turn >= len(task.reference):
+            return "Done.", []
+        step = task.reference[turn]
+        if "calls" in step:
+            return "", [RawToolCall(c["tool"], dict(c.get("args", {}))) for c in step["calls"]]
+        return str(step["final"]), []
+
+    def _step(self, task: Task, turn: int) -> tuple[str, list[RawToolCall]]:
+        profile = self.profile
+        if profile == "oracle":
+            return self._oracle(task, turn)
+        if profile == "call_limit":
+            return "", [RawToolCall("list_dir", {"path": "."})]
+        if profile == "no_call":
+            return "I believe the answer is 42.", []
+        if profile == "bad_edit":
+            content, calls = self._oracle(task, turn)
+            for call in calls:
+                if call.name == "edit_file":
+                    call.arguments = {**call.arguments, "old_text": "THIS TEXT IS NOT IN THE FILE"}
+                elif call.name == "apply_patch":
+                    call.arguments = {"patch": "--- a/missing.py\n+++ b/missing.py\n@@ -1,1 +1,1 @@\n-x\n+y\n"}
+            return content, calls
+        if turn >= 1:
+            return "Done.", []
+        first = next((c for step in task.reference for c in step.get("calls", [])), None)
+        if first is None:
+            # The task needs no tools. A wrong_tool model calls one anyway; the others answer correctly.
+            if profile == "wrong_tool":
+                return "", [RawToolCall("list_dir", {})]
+            return self._oracle(task, turn)
+        tool, args = first["tool"], dict(first.get("args", {}))
+        if profile == "invalid_json":
+            return "", [RawToolCall(tool, '{"path": ')]
+        if profile == "unknown_tool":
+            return "", [RawToolCall("search_everything", {"query": "x"})]
+        if profile == "wrong_tool":
+            return "", [RawToolCall("list_dir", {})]
+        if profile == "text_call":
+            return json.dumps({"name": tool, "arguments": args}), []
+        raise AssertionError(profile)  # pragma: no cover