halfabyte-blackbox 0.1.0__tar.gz

halfabyte_blackbox-0.1.0/PKG-INFO

@@ -0,0 +1,135 @@
+Metadata-Version: 2.4
+Name: halfabyte-blackbox
+Version: 0.1.0
+Summary: A flight recorder for AI agents: record every step, find the step that caused a failure, prove it by replay, and fix it without regressions.
+Author: Aryan Lomte, Radhesh, Aditya, Advay Chavan
+License: MIT
+Keywords: ai agents,llm,observability,debugging,replay,root cause analysis,mcp,evaluation
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Debuggers
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: pydantic>=2.5
+Requires-Dist: pyyaml>=6
+Requires-Dist: python-dotenv>=1
+Requires-Dist: openai>=1.40
+Provides-Extra: checker
+Requires-Dist: laya; extra == "checker"
+Requires-Dist: system-one-adapter[openai]; extra == "checker"
+Provides-Extra: jev
+Requires-Dist: typesafe-sdk; extra == "jev"
+Provides-Extra: ml
+Requires-Dist: lightgbm>=4; extra == "ml"
+Requires-Dist: scikit-learn>=1.4; extra == "ml"
+Requires-Dist: shap>=0.45; extra == "ml"
+Requires-Dist: sentence-transformers>=3; extra == "ml"
+Requires-Dist: numpy; extra == "ml"
+Provides-Extra: ui
+Requires-Dist: fastapi>=0.110; extra == "ui"
+Requires-Dist: uvicorn[standard]>=0.29; extra == "ui"
+Requires-Dist: sse-starlette>=2; extra == "ui"
+Provides-Extra: agents
+Requires-Dist: rank-bm25>=0.2; extra == "agents"
+Provides-Extra: data
+Requires-Dist: datasets>=2.18; extra == "data"
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Requires-Dist: ruff>=0.5; extra == "dev"
+Provides-Extra: all
+Requires-Dist: halfabyte-blackbox[agents,checker,data,ml,ui]; extra == "all"
+
+# Black Box: a flight recorder for AI agents
+
+When an AI agent fails, the mistake usually happened several steps before the wrong answer. Black Box records
+every step of an agent run, **finds the step that caused the failure, and proves it**: it re-runs the agent from
+its recording with only that step repaired. If the run now passes, that step was the cause. Unchanged steps come
+from the recording, so a replay costs zero model calls and a fix re-runs only what changed.
+
+```bash
+pip install halfabyte-blackbox                 # recorder, replay, proof, CLI, CI, MCP server, trace import
+pip install "halfabyte-blackbox[ui]"           # + the API / dashboard backend
+pip install "halfabyte-blackbox[all]"          # + step checker, ranking model, test agents, dataset loaders
+```
+
+## Add it to an existing agent (3 lines)
+
+```python
+import blackbox
+from openai import OpenAI
+
+client = blackbox.wrap(OpenAI())             # 1. every model call is recorded
+
+@blackbox.tool                               # 2. every tool call is recorded and replayable
+def get_weather(city: str) -> dict:
+    ...
+
+def my_agent(question: str) -> str:          # your agent, unchanged
+    ...
+
+trace = blackbox.run(my_agent, "Will it rain in Pune?")       # 3. run it under the recorder
+```
+
+```python
+blackbox.replay(trace)                                         # re-run from the recording: 0 model calls
+child = blackbox.fork(trace, 2, blackbox.Change(kind="output", value={"result": {"rain": True}}))
+blackbox.savings(child)                                        # steps re-run vs re-used, tokens saved
+```
+
+Works with agents you did not write (shown on Hugging Face smolagents), and with traces you already export:
+`blackbox import otel traces.json` / `blackbox import langfuse trace.json`.
+
+## What it does
+
+| Capability | How |
+|---|---|
+| **Record** every model call, tool call, search and memory change, with a save-point after each step | `blackbox.wrap`, `@blackbox.tool`, `blackbox.run` |
+| **Diagnose**: rank the steps most likely to have caused a failure, with plain-language evidence | `blackbox show <run>`, API `/api/runs/{id}/diagnosis` |
+| **Prove** the cause by re-running with one suspect repaired at a time | Verify (API, MCP `verify`) |
+| **Causal report**: necessary vs sufficient, joint causes, every recovery path, blast radius | API `/causal`, MCP `causal_report` |
+| **One fix for many**: test one rule on similar past failures and passing runs; APPROVE only if nothing breaks | `blackbox fleet <run>` |
+| **Crash test**: plant every known kind of mistake into a working agent, grade it 1–5 stars | `blackbox crash-test <run>` |
+| **Seen this before?**: failure fingerprints, look-alike failures, novel failures, known fixes | MCP `similar_failures` |
+| **Guardian**: incidents in plain words on email, WhatsApp, Slack, Telegram; money-moving agents paused until approved | `blackbox.Guardian(...)`, `blackbox guardian test` |
+| **Black Box CI**: replay pinned recorded runs on every pull request; fail on regressions | `blackbox ci pin` / `blackbox ci run`, GitHub Action |
+| **MCP server**: let Claude Code, Cursor or VS Code investigate failures with 28 tools | `blackbox mcp` |
+| **Live Lab**: ask any question, plant a mistake live, watch the 14-stage investigation | `blackbox lab "question"` |
+
+Safety: tools that move money never execute during any re-run, and Guardian never retries them without a human.
+
+## Command line
+
+```bash
+blackbox runs --fail             # failed runs
+blackbox show <run_id>           # step by step, with who-used-whose-output links
+blackbox replay <run_id>         # replay from the recording
+blackbox crash-test <run_id>     # red-team a passing run
+blackbox fleet <run_id> --rule "..."   # one fix for many, with a regression firewall
+blackbox ci pin && blackbox ci run      # regression firewall for agent code
+blackbox guardian channels       # which alert channels are configured
+blackbox mcp                     # MCP server on stdio
+blackbox ui                      # API on http://127.0.0.1:8000
+```
+
+Models are named in one file, `models.yaml` (Ollama locally, Groq or OpenRouter hosted); keys live in `.env`.
+
+## Development (this repository)
+
+```bash
+pip install -e ".[all,dev]"
+py -3 -m pytest -q -m "not live"
+cd web && npm install && npm run dev       # dashboard on http://localhost:5173
+```
+
+Design: `SYSTEM.md` · rules: `CLAUDE.md` · UI contract: `docs/API_FOR_UI.md` · MCP: `docs/MCP.md` · CI: `docs/CI.md` ·
+Guardian setup: `docs/GUARDIAN_SETUP.md`. Branches `<name>/<feature>` → PR into `dev` → `main` at milestones.
+
+Built by team Half a Byte: Aryan Lomte, Radhesh, Aditya, Advay Chavan.

halfabyte_blackbox-0.1.0/README.md

@@ -0,0 +1,86 @@
+# Black Box: a flight recorder for AI agents
+
+When an AI agent fails, the mistake usually happened several steps before the wrong answer. Black Box records
+every step of an agent run, **finds the step that caused the failure, and proves it**: it re-runs the agent from
+its recording with only that step repaired. If the run now passes, that step was the cause. Unchanged steps come
+from the recording, so a replay costs zero model calls and a fix re-runs only what changed.
+
+```bash
+pip install halfabyte-blackbox                 # recorder, replay, proof, CLI, CI, MCP server, trace import
+pip install "halfabyte-blackbox[ui]"           # + the API / dashboard backend
+pip install "halfabyte-blackbox[all]"          # + step checker, ranking model, test agents, dataset loaders
+```
+
+## Add it to an existing agent (3 lines)
+
+```python
+import blackbox
+from openai import OpenAI
+
+client = blackbox.wrap(OpenAI())             # 1. every model call is recorded
+
+@blackbox.tool                               # 2. every tool call is recorded and replayable
+def get_weather(city: str) -> dict:
+    ...
+
+def my_agent(question: str) -> str:          # your agent, unchanged
+    ...
+
+trace = blackbox.run(my_agent, "Will it rain in Pune?")       # 3. run it under the recorder
+```
+
+```python
+blackbox.replay(trace)                                         # re-run from the recording: 0 model calls
+child = blackbox.fork(trace, 2, blackbox.Change(kind="output", value={"result": {"rain": True}}))
+blackbox.savings(child)                                        # steps re-run vs re-used, tokens saved
+```
+
+Works with agents you did not write (shown on Hugging Face smolagents), and with traces you already export:
+`blackbox import otel traces.json` / `blackbox import langfuse trace.json`.
+
+## What it does
+
+| Capability | How |
+|---|---|
+| **Record** every model call, tool call, search and memory change, with a save-point after each step | `blackbox.wrap`, `@blackbox.tool`, `blackbox.run` |
+| **Diagnose**: rank the steps most likely to have caused a failure, with plain-language evidence | `blackbox show <run>`, API `/api/runs/{id}/diagnosis` |
+| **Prove** the cause by re-running with one suspect repaired at a time | Verify (API, MCP `verify`) |
+| **Causal report**: necessary vs sufficient, joint causes, every recovery path, blast radius | API `/causal`, MCP `causal_report` |
+| **One fix for many**: test one rule on similar past failures and passing runs; APPROVE only if nothing breaks | `blackbox fleet <run>` |
+| **Crash test**: plant every known kind of mistake into a working agent, grade it 1–5 stars | `blackbox crash-test <run>` |
+| **Seen this before?**: failure fingerprints, look-alike failures, novel failures, known fixes | MCP `similar_failures` |
+| **Guardian**: incidents in plain words on email, WhatsApp, Slack, Telegram; money-moving agents paused until approved | `blackbox.Guardian(...)`, `blackbox guardian test` |
+| **Black Box CI**: replay pinned recorded runs on every pull request; fail on regressions | `blackbox ci pin` / `blackbox ci run`, GitHub Action |
+| **MCP server**: let Claude Code, Cursor or VS Code investigate failures with 28 tools | `blackbox mcp` |
+| **Live Lab**: ask any question, plant a mistake live, watch the 14-stage investigation | `blackbox lab "question"` |
+
+Safety: tools that move money never execute during any re-run, and Guardian never retries them without a human.
+
+## Command line
+
+```bash
+blackbox runs --fail             # failed runs
+blackbox show <run_id>           # step by step, with who-used-whose-output links
+blackbox replay <run_id>         # replay from the recording
+blackbox crash-test <run_id>     # red-team a passing run
+blackbox fleet <run_id> --rule "..."   # one fix for many, with a regression firewall
+blackbox ci pin && blackbox ci run      # regression firewall for agent code
+blackbox guardian channels       # which alert channels are configured
+blackbox mcp                     # MCP server on stdio
+blackbox ui                      # API on http://127.0.0.1:8000
+```
+
+Models are named in one file, `models.yaml` (Ollama locally, Groq or OpenRouter hosted); keys live in `.env`.
+
+## Development (this repository)
+
+```bash
+pip install -e ".[all,dev]"
+py -3 -m pytest -q -m "not live"
+cd web && npm install && npm run dev       # dashboard on http://localhost:5173
+```
+
+Design: `SYSTEM.md` · rules: `CLAUDE.md` · UI contract: `docs/API_FOR_UI.md` · MCP: `docs/MCP.md` · CI: `docs/CI.md` ·
+Guardian setup: `docs/GUARDIAN_SETUP.md`. Branches `<name>/<feature>` → PR into `dev` → `main` at milestones.
+
+Built by team Half a Byte: Aryan Lomte, Radhesh, Aditya, Advay Chavan.

halfabyte_blackbox-0.1.0/blackbox/__init__.py

@@ -0,0 +1,36 @@
+"""Black Box: a flight recorder for AI agents.
+
+Quick start for an existing project (see README.md):
+
+    import blackbox
+    client = blackbox.wrap(OpenAI())
+    trace = blackbox.run(my_agent, "question")
+"""
+
+from blackbox.agents.base import Agent, Task, register_agent, register_checker, run_task
+from blackbox.core.schema import Change, Outcome, Run, Step, Trace
+from blackbox.guardian import Guardian, Incident
+from blackbox.core.store import Store
+from blackbox.integrations import (
+    Session,
+    afork,
+    areplay,
+    arun,
+    configure,
+    default_store,
+    retriever,
+    run,
+    session,
+    tool,
+    wrap,
+)
+from blackbox.recorder.context import RunContext, ToolError
+from blackbox.replay.engine import fork, replay, savings
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "Agent", "Change", "Guardian", "Incident", "afork", "areplay", "arun", "Outcome", "Run", "RunContext", "Session", "Step", "Store", "Task", "ToolError",
+    "Trace", "configure", "default_store", "fork", "register_agent", "register_checker", "replay",
+    "retriever", "run", "run_task", "savings", "session", "tool", "wrap",
+]

halfabyte_blackbox-0.1.0/blackbox/agents/base.py

@@ -0,0 +1,223 @@
+"""Agents are resumable state machines (CLAUDE.md hard rule 3).
+
+    class QAAgent(Agent):
+        name, version, family, entry = "qa_agent", "v1", "qa", "plan"
+        def init_state(self, task): return {"question": task.input["question"]}
+        def plan(self, ctx):   ...; return "search"     # next node name
+        def search(self, ctx): ...; return "answer"
+        def answer(self, ctx): ctx.final(...); return None   # None = finished
+        nodes = {"plan": plan, "search": search, "answer": answer}
+
+Rules:
+- Every model/tool/search call goes through ctx (ctx.llm / ctx.tool / ctx.retrieve).
+- All memory lives in ctx.state; assign values (state[k] = v), never mutate in place.
+- Randomness: ctx.rng. Time: ctx.now(). Nothing else.
+- Finish by calling ctx.final(answer) and returning None.
+"""
+
+from __future__ import annotations
+
+import os
+import socket
+import traceback
+import uuid
+from dataclasses import dataclass, field
+from typing import Any, Callable, ClassVar
+
+from blackbox.core.schema import Change, Outcome, Run, Trace
+from blackbox.core.store import Store
+from blackbox.recorder.context import AutopilotRewind, Monitor, ReplayPlan, RunContext, StepListener, current
+
+MAX_NODE_VISITS = 60
+
+Checker = Callable[[Any, Any], Outcome]  # (final_answer, expected) -> Outcome
+
+
+@dataclass
+class Task:
+    family: str
+    task_id: str
+    input: dict  # what the agent may see
+    expected: Any  # what the checker compares against; the agent never sees it
+    meta: dict = field(default_factory=dict)
+
+
+class Agent:
+    name: ClassVar[str] = "base"
+    version: ClassVar[str] = "v1"
+    family: ClassVar[str] = ""
+    entry: ClassVar[str] = ""
+    nodes: ClassVar[dict[str, Callable[["Agent", RunContext], str | None]]] = {}
+
+    def init_state(self, task: Task) -> dict:
+        return {}
+
+    def tool_caller(self, task: Task) -> Callable[[str, dict], Any] | None:
+        return None
+
+    def retriever(self, task: Task) -> Callable[..., list[dict]] | None:
+        return None
+
+    @classmethod
+    def ref(cls) -> str:
+        return f"{cls.name}@{cls.version}"
+
+
+# ---- registries (agents and checkers are looked up by name on replay) ----------
+_AGENTS: dict[str, type[Agent]] = {}
+_CHECKERS: dict[str, Checker] = {}
+
+
+def register_agent(cls: type[Agent]) -> type[Agent]:
+    _AGENTS[cls.ref()] = cls
+    return cls
+
+
+def register_checker(family: str, fn: Checker) -> Checker:
+    _CHECKERS[family] = fn
+    return fn
+
+
+def _load_builtin() -> None:
+    """Register our test agents on first miss, so a process that only replays
+    stored runs (API, CLI) finds them. Skipped when the `agents` extra is absent."""
+    try:
+        import blackbox.agents.builtin  # noqa: F401
+    except ModuleNotFoundError as exc:
+        if exc.name is None or exc.name.startswith("blackbox"):
+            raise
+
+
+def get_agent(ref: str) -> Agent:
+    if ref not in _AGENTS and ref.endswith("@fn"):
+        from blackbox.integrations import resolve_function_agent  # user function agents
+
+        resolve_function_agent(ref)
+    if ref not in _AGENTS:
+        _load_builtin()
+    if ref not in _AGENTS:
+        raise KeyError(f"agent {ref!r} not registered (import its module). Known: {sorted(_AGENTS)}")
+    return _AGENTS[ref]()
+
+
+def get_checker(family: str) -> Checker | None:
+    """None for a user's project that registered no checker: the run is recorded
+    without a pass/fail verdict rather than with an invented one."""
+    return _CHECKERS.get(family)
+
+
+def machine_name() -> str:
+    return os.environ.get("BLACKBOX_MACHINE") or socket.gethostname()
+
+
+def run_task(
+    agent: Agent,
+    task: Task,
+    model_id: str,
+    *,
+    store: Store | None = None,
+    plan: ReplayPlan | None = None,
+    monitor: Monitor | None = None,
+    on_step: list[StepListener] | None = None,
+    run_id: str | None = None,
+    parent_run_id: str | None = None,
+    fork_step_idx: int | None = None,
+    patch: Change | None = None,
+    split: str | None = None,
+    autopilot=None,
+) -> Trace:
+    """Run (or replay/fork) one task and return its trace. Saved if a store is given.
+
+    A crash (exception escaping the agent) marks the run "crashed" with the
+    traceback. It is our bug, never a task failure, and never a label
+    (CLAUDE.md review rule 3).
+    """
+    run = Run(
+        run_id=run_id or uuid.uuid4().hex[:16],
+        task_family=task.family,
+        task_id=task.task_id,
+        task_input=task.input,
+        agent=agent.ref(),
+        model_id=model_id,
+        parent_run_id=parent_run_id,
+        fork_step_idx=fork_step_idx,
+        patch=patch,
+        split=split,
+        machine=machine_name(),
+        task_expected=task.expected,
+    )
+    ctx = RunContext(
+        run, agent.init_state(task), store=store,
+        tool_caller=agent.tool_caller(task), retriever=agent.retriever(task),
+        plan=plan, monitor=monitor, on_step=on_step, autopilot=autopilot,
+    )
+    token = current.set(ctx)
+    try:
+        node: str | None = agent.entry
+        visits = 0
+        while node is not None:
+            if node not in agent.nodes:
+                raise KeyError(f"{agent.ref()}: unknown node {node!r}")
+            visits += 1
+            if visits > MAX_NODE_VISITS:
+                # Looping forever is a genuine agent failure, not a crash.
+                run.stats["stopped"] = f"exceeded {MAX_NODE_VISITS} node visits"
+                if not ctx.steps or ctx.steps[-1].kind != "final":
+                    ctx.final(None)
+                break
+            ctx.node = node
+            node = agent.nodes[node](agent, ctx)
+        ctx.close()
+        final_steps = [s for s in ctx.steps if s.kind == "final"]
+        answer = final_steps[-1].output["answer"] if final_steps else None
+        checker = get_checker(task.family)
+        run.outcome = checker(answer, task.expected) if checker else None
+        run.status = "done"
+    except AutopilotRewind as rewind:
+        current.reset(token)
+        token = None
+        return _take_over(agent, task, model_id, ctx, run, rewind, store=store, monitor=monitor,
+                          on_step=on_step, split=split, autopilot=autopilot)
+    except Exception:  # noqa: BLE001 - recorded loudly as a crash, see docstring
+        ctx.close(monitor_timeout=5.0)
+        run.status = "crashed"
+        run.error = traceback.format_exc()
+    finally:
+        if token is not None:
+            current.reset(token)
+    run.stats = {**ctx.stats(), **run.stats}
+    trace = Trace(run=run, steps=ctx.steps)
+    if store is not None:
+        store.save_trace(trace)
+    return trace
+
+
+def _take_over(agent, task, model_id, ctx, run, rewind: AutopilotRewind, *, store, monitor, on_step, split,
+               autopilot) -> Trace:
+    """Autopilot (USP X1): the run is stopped before it can answer. Its steps so
+    far are kept as a 'rewound' run, and a fork from the flagged decision takes
+    over, re-using everything before it from the recording."""
+    ctx.close(monitor_timeout=5.0)
+    flagged = ctx.steps[rewind.step_idx]
+    flagged.monitor = rewind.answers if rewind.answers is not None else flagged.monitor
+    run.status = "rewound"
+    run.stats = {**ctx.stats(), "autopilot": {"flagged_step": rewind.step_idx, "rewound_to": rewind.target_idx}}
+    partial = Trace(run=run, steps=ctx.steps)
+    if store is not None:
+        store.save_trace(partial)
+    child = run_task(
+        agent, task, model_id, store=store, monitor=monitor, on_step=on_step, split=split,
+        plan=ReplayPlan(parent=partial, fork_idx=rewind.target_idx, change=rewind.change),
+        parent_run_id=run.run_id, fork_step_idx=rewind.target_idx, patch=rewind.change, autopilot=autopilot,
+    )
+    child.run.stats.setdefault("autopilot_rescue", {
+        "from_run": run.run_id, "flagged_step": rewind.step_idx, "rewound_to": rewind.target_idx,
+        "change": rewind.change.model_dump(mode="json") if rewind.change else None,
+    })
+    if store is not None:
+        store.save_trace(child)
+    return child
+
+
+def task_from_run(run: Run) -> Task:
+    return Task(family=run.task_family, task_id=run.task_id, input=run.task_input, expected=run.task_expected)

halfabyte_blackbox-0.1.0/blackbox/agents/builtin.py

@@ -0,0 +1,7 @@
+"""Importing this module registers our test agents and their checkers.
+
+Replay looks agents up by name (`qa_agent@v1`), so any process that replays
+or forks a stored run (farm, API, CLI) needs them registered first.
+"""
+
+from blackbox.agents import checkers, math_agent, qa, shop  # noqa: F401 - registration side effect

halfabyte_blackbox-0.1.0/blackbox/agents/checkers.py

@@ -0,0 +1,83 @@
+"""Pass/fail checkers. Plain code: no model decides what counts as a failure
+(SYSTEM.md §3.1, CLAUDE.md hard rule 5).
+
+    QA   : SQuAD-normalised word-overlap F1 >= 0.6 against the gold answer
+    Math : the final number equals the gold number exactly
+"""
+
+from __future__ import annotations
+
+import re
+import string
+from collections import Counter
+from typing import Any
+
+from blackbox.agents.base import register_checker
+from blackbox.core.schema import Outcome
+
+QA_F1_THRESHOLD = 0.6
+
+_ARTICLES = re.compile(r"\b(a|an|the)\b")
+_PUNCT = str.maketrans("", "", string.punctuation)
+_NUMBER = re.compile(r"-?\d[\d,]*\.?\d*|-?\.\d+")
+
+
+def normalize_answer(text: Any) -> str:
+    """SQuAD normalisation: lowercase, drop punctuation, drop articles, squeeze spaces."""
+    s = "" if text is None else str(text)
+    s = s.lower().translate(_PUNCT)
+    s = _ARTICLES.sub(" ", s)
+    return " ".join(s.split())
+
+
+def f1_score(prediction: Any, gold: Any) -> float:
+    pred, ref = normalize_answer(prediction).split(), normalize_answer(gold).split()
+    if not pred or not ref:
+        return float(pred == ref)
+    common = sum((Counter(pred) & Counter(ref)).values())
+    if common == 0:
+        return 0.0
+    precision, recall = common / len(pred), common / len(ref)
+    return 2 * precision * recall / (precision + recall)
+
+
+def _gold(expected: Any) -> Any:
+    return expected.get("answer") if isinstance(expected, dict) else expected
+
+
+def check_qa(answer: Any, expected: Any) -> Outcome:
+    gold = _gold(expected)
+    score = f1_score(answer, gold)
+    # yes/no questions: "yes, they are" must not pass on partial overlap, nor "no" match "yes"
+    if normalize_answer(gold) in ("yes", "no"):
+        words = normalize_answer(answer).split()
+        score = float(bool(words) and words[0] == normalize_answer(gold))
+    return Outcome(success=score >= QA_F1_THRESHOLD, score=round(score, 4),
+                   final_answer=answer, expected=gold, checker="qa_f1")
+
+
+def parse_number(value: Any) -> float | None:
+    """The last number in the text ("So she has $1,250.00 left." -> 1250.0)."""
+    if isinstance(value, bool) or value is None:
+        return None
+    if isinstance(value, (int, float)):
+        return float(value)
+    found = _NUMBER.findall(str(value))
+    for raw in reversed(found):
+        cleaned = raw.replace(",", "").rstrip(".")
+        try:
+            return float(cleaned)
+        except ValueError:
+            continue
+    return None
+
+
+def check_math(answer: Any, expected: Any) -> Outcome:
+    gold = parse_number(_gold(expected))
+    got = parse_number(answer)
+    ok = gold is not None and got is not None and abs(got - gold) < 1e-6
+    return Outcome(success=ok, score=float(ok), final_answer=answer, expected=_gold(expected), checker="math_exact")
+
+
+register_checker("qa", check_qa)
+register_checker("math", check_math)

halfabyte_blackbox-0.1.0/blackbox/agents/math_agent.py

@@ -0,0 +1,104 @@
+"""Math agent for GSM8K: plan -> (step -> calculate)* -> check -> answer.
+
+The model never does arithmetic itself: it names one calculation at a time
+and the calculator tool computes it. A run is 6 to 15 steps:
+
+    plan(llm)  [step(llm)  calculator(tool)] x 1..MAX_CALCS  step(llm, DONE)  check(llm)  final
+"""
+
+from __future__ import annotations
+
+from blackbox.agents import checkers  # noqa: F401 - registers the "math" checker
+from blackbox.agents.base import Agent, Task, register_agent
+from blackbox.agents.parsing import clip, field
+from blackbox.agents.tools import call_tool
+from blackbox.recorder.context import RunContext, ToolError
+
+MAX_CALCS = 5
+MAX_TURNS = 6  # step-node visits, including the one that says DONE
+
+PLAN_SYSTEM = (
+    "You plan how to solve a math word problem. List the calculations needed, in order, "
+    "one short line each: what is being found and from which numbers in the problem. "
+    "Do not compute the results. At most 5 lines. The last line must find what the question asks."
+)
+STEP_SYSTEM = (
+    "You solve a math word problem one calculation at a time. A calculator does the arithmetic. "
+    "Do the next line of the plan that has no result yet. Reply with exactly one line:\n"
+    "CALC: <one arithmetic expression using only numbers and + - * / ( )>\n"
+    "or, when every line of the plan has a result:\nDONE"
+)
+CHECK_SYSTEM = (
+    "You check the solution of a math word problem. Given the problem and the calculator results, "
+    "state the final answer to the question asked. Reply with exactly one line:\n"
+    "FINAL: <number only, no units>"
+)
+
+
+def _format_results(results: list[dict]) -> str:
+    if not results:
+        return "(none yet)"
+    return "\n".join(f"{i + 1}. {r['expr']} = {r['value']}" for i, r in enumerate(results))
+
+
+@register_agent
+class MathAgent(Agent):
+    name, version, family, entry = "math_agent", "v1", "math", "plan"
+
+    def init_state(self, task: Task) -> dict:
+        return {"question": task.input["question"], "results": [], "turns": 0}
+
+    def tool_caller(self, task: Task):
+        return call_tool
+
+    def plan(self, ctx: RunContext) -> str:
+        res = ctx.llm(ctx.model_id, [
+            {"role": "system", "content": PLAN_SYSTEM},
+            {"role": "user", "content": f"Problem: {ctx.state['question']}"},
+        ], max_tokens=160)
+        ctx.state["plan"] = clip(res.text, 600)
+        return "step"
+
+    def step(self, ctx: RunContext) -> str:
+        results, turns = ctx.state["results"], ctx.state["turns"]
+        res = ctx.llm(ctx.model_id, [
+            {"role": "system", "content": STEP_SYSTEM},
+            {"role": "user", "content": (
+                f"Problem: {ctx.state['question']}\n\nPlan:\n{ctx.state['plan']}\n\n"
+                f"Results so far:\n{_format_results(results)}"
+            )},
+        ], max_tokens=48)
+        expr = field(res.text, "CALC")
+        ctx.state["turns"] = turns + 1
+        if expr is None or len(results) >= MAX_CALCS or turns + 1 >= MAX_TURNS:
+            return "check"
+        ctx.state["expr"] = expr
+        return "calculate"
+
+    def calculate(self, ctx: RunContext) -> str:
+        expr = ctx.state["expr"]
+        try:
+            value = ctx.tool("calculator", expression=expr)
+        except ToolError as exc:
+            value = f"error: {clip(str(exc), 120)}"  # recorded on the step; the model sees it next turn
+        # read after the call: the calculator depends on `expr` only, so its
+        # parents and its re-use fingerprint must not include earlier results
+        ctx.state["results"] = ctx.state["results"] + [{"expr": expr, "value": value}]
+        return "step"
+
+    def check(self, ctx: RunContext) -> str:
+        results = ctx.state["results"]
+        res = ctx.llm(ctx.model_id, [
+            {"role": "system", "content": CHECK_SYSTEM},
+            {"role": "user", "content": (
+                f"Problem: {ctx.state['question']}\n\nCalculator results:\n{_format_results(results)}"
+            )},
+        ], max_tokens=32)
+        ctx.state["answer"] = field(res.text, "FINAL") or clip(res.text, 80)
+        return "answer"
+
+    def answer(self, ctx: RunContext) -> None:
+        ctx.final(ctx.state["answer"])
+        return None
+
+    nodes = {"plan": plan, "step": step, "calculate": calculate, "check": check, "answer": answer}