agentfork-cli 0.2.0__tar.gz → 0.3.0__tar.gz

{agentfork_cli-0.2.0 → agentfork_cli-0.3.0}/PKG-INFO

@@ -1,13 +1,13 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: agentfork-cli
-Version: 0.2.0
+Version: 0.3.0
 Summary: Fork your LLM agent at any step and sweep prompts, models or tool results to find what fixes it.
 Project-URL: Homepage, https://github.com/bimal1023/agentfork
 Project-URL: Issues, https://github.com/bimal1023/agentfork/issues
 Author: Bimal Kumal
 License-Expression: MIT
 License-File: LICENSE
-Keywords: agents,anthropic,debugging,llm,openai,replay,testing
+Keywords: agents,anthropic,debugging,langgraph,llm,openai,replay,testing
 Classifier: Development Status :: 3 - Alpha
 Classifier: Intended Audience :: Developers
 Classifier: Programming Language :: Python :: 3
@@ -22,6 +22,8 @@ Provides-Extra: dev
 Requires-Dist: anthropic; extra == 'dev'
 Requires-Dist: fastapi; extra == 'dev'
 Requires-Dist: httpx; extra == 'dev'
+Requires-Dist: langchain-openai; extra == 'dev'
+Requires-Dist: langgraph; extra == 'dev'
 Requires-Dist: openai; extra == 'dev'
 Requires-Dist: pytest; extra == 'dev'
 Requires-Dist: uvicorn; extra == 'dev'
@@ -49,28 +51,30 @@ Everything runs locally. No account, no server, no data leaves your machine.
 A refund agent reads `"amount": 4999, "currency": "USD"` from an order API. The amount is in cents, and the agent refunds **$4,999** for a $49.99 laptop ([examples/refund_agent_cents.py](examples/refund_agent_cents.py)). Is the fix a better model or a better prompt?
 
 ```text
-$ agentfork sweep de994fe6 --step 3 --repeat 3 --expect "49.99" --model gpt-4o-mini,gpt-4o
+$ agentfork sweep 543da612 --step 3 --repeat 3 --expect "49.99" --model gpt-4o-mini,gpt-4o
+Estimated cost: about $0.0050
 
-variant             runs  pass  avg tokens  sample output
-original            1     0/1   445         I have processed a refund of $4,999.00...
-model=gpt-4o-mini   3     0/3   444         I have processed a refund of $4,999.00...
-model=gpt-4o        3     0/3   292         I apologize, it seems there was a mis...
+  variant             ok    pass   tokens   time      cost   sample output
+  original            1/1   0/1       446   1.7s   $0.0001   I have processed a refund of $4,999.00...
+  model=gpt-4o-mini   3/3   0/3       446   1.4s   $0.0001   I have processed a refund of $4,999.00...
+  model=gpt-4o        3/3   0/3       394   1.2s   $0.0015   The full amount of $4,999.00 for order A123...
 ```
 
-A bigger model doesn't fix it. One sentence in the prompt does:
+A bigger model costs 15x more per run and doesn't fix it. One sentence in the prompt does:
 
 ```text
-$ agentfork sweep de994fe6 --step 3 --repeat 3 --expect "49.99" \
+$ agentfork sweep 543da612 --step 3 --repeat 3 --expect "49.99" \
     --system "You are a support agent. Look up the order, then refund the full order amount in USD." \
     --system "... Order amounts from the API are in cents; divide by 100 before refunding."
+Estimated cost: about $0.0006
 
-variant      runs  pass  avg tokens  sample output
-original     1     0/1   445         I have processed a refund of $4,999.00...
-system #1    3     0/3   445         A full refund of $4,999.00 has ...
-system #2    3     3/3   460         I have processed a refund of $49.99...
+  variant     ok    pass   tokens   time      cost   sample output
+  original    1/1   0/1       446   1.7s   $0.0001   I have processed a refund of $4,999.00...
+  system #1   3/3   0/3       445   1.2s   $0.0001   I've successfully refunded you $4,999.00...
+  system #2   3/3   3/3       457   1.2s   $0.0001   I have processed a refund of $49.99...
 ```
 
-Steps 1 and 2 replayed from cache in every run, so only the step that mattered was paid for.
+Steps 1 and 2 replayed from cache in every run, so only the step that mattered was paid for, and the runs went 4 at a time.
 
 ## Quickstart
 
@@ -95,6 +99,33 @@ def run_agent(question): ...
 
 Run your agent as usual. Each run is saved to `.agentfork/traces.db`.
 
+### LangGraph
+
+Wrap the chat model and put `@agentfork.tool` under LangChain's `@tool`. Forks and sweeps then work on the graph like on any other agent ([examples/langgraph_refund_agent.py](examples/langgraph_refund_agent.py)):
+
+```python
+import agentfork
+from langchain_core.tools import tool
+from langchain_openai import ChatOpenAI
+from langgraph.prebuilt import create_react_agent
+
+llm = agentfork.wrap_langchain(ChatOpenAI(model="gpt-4o-mini"))   # or ChatAnthropic
+
+@tool
+@agentfork.tool
+def lookup_order(order_id: str) -> str:
+    """Look up an order by id."""
+    ...
+
+graph = create_react_agent(llm, [lookup_order])
+
+@agentfork.trace
+def run_agent(question):
+    return graph.invoke({"messages": [("user", question)]})["messages"][-1].content
+```
+
+Streaming models (`streaming=True`) and the OpenAI Responses API aren't recorded yet.
+
 ## Commands
 
 | Command | What it does |
@@ -102,7 +133,7 @@ Run your agent as usual. Each run is saved to `.agentfork/traces.db`.
 | `agentfork list` | Recent runs with steps, tokens, duration and fork lineage |
 | `agentfork show <id> [--full]` | A run as a step tree, with optional full JSON |
 | `agentfork fork <id> --step N --output/--model/--system ...` | Re-run from step N with one change; earlier steps come from cache |
-| `agentfork sweep <id> --step N --model a,b --system ... --repeat K --expect TEXT` | Try many variants of step N, K times each, and compare pass rate, tokens and time |
+| `agentfork sweep <id> --step N --model a,b --system ... --repeat K --expect TEXT [--parallel P]` | Try many variants of step N, K times each, P at a time (default 4), and compare pass rate, tokens, time and cost |
 | `agentfork diff <a> <b>` | Compare two runs step by step |
 | `agentfork save-test <id>` | Save a run as a pytest test that replays from cache in CI with no API key |
 | `agentfork ui [--port N]` | Open the local web UI to browse, fork, sweep and compare runs |
@@ -133,11 +164,19 @@ Start it from the folder you ran your agent in, since runs are saved to `.agentf
 
 ## How it works
 
-1. **Record.** `@trace`, `@tool` and `wrap_openai`/`wrap_anthropic` capture each step's inputs, outputs, tokens and latency, plus a hash of its inputs.
+1. **Record.** `@trace`, `@tool` and `wrap_openai`/`wrap_anthropic`/`wrap_langchain` capture each step's inputs, outputs, tokens and latency, plus a hash of its inputs.
 2. **Fork.** agentfork re-imports your agent and re-runs it. Steps before N are served from the cache, step N gets your change, and everything after runs live. A model change applies to every LLM call from step N on.
-3. **Sweep.** Runs fork for each variant `--repeat` times and checks each final output for your `--expect` text.
+3. **Sweep.** Runs fork for each variant `--repeat` times, `--parallel` at a time, and checks each final output for your `--expect` text. Before it starts, it estimates the cost from the tokens the original run used from step N on, priced per model.
 4. **Test.** `save-test` writes a JSON snapshot plus a pytest file. The test replays every step from cache and fails with a diff if any step's inputs, the step count, or the final output changes.
 
+## Costs
+
+Costs use list prices per model for OpenAI and Anthropic, built into [pricing.py](src/agentfork/pricing.py). Prices change, and a model without a price shows `-`. To add or correct one, put USD per 1M input and output tokens in `.agentfork/prices.json`:
+
+```json
+{"gpt-4o-mini": [0.15, 0.60], "my-finetune": [3.0, 12.0]}
+```
+
 ## Related work
 
 agentfork overlaps with record/replay tools like [agent-rewind](https://pypi.org/project/agent-rewind/), which supports many more providers, and [agentstep](https://pypi.org/project/agentstep/), a LangGraph time-travel debugger with a web UI. agentfork's focus is the step after replay: sweeping many variants of one step, with repeats and pass rates, to choose a fix with evidence.
@@ -148,9 +187,9 @@ agentfork overlaps with record/replay tools like [agent-rewind](https://pypi.org
 - [x] OpenAI and Anthropic support
 - [x] Sweep with repeats and pass rates
 - [x] Local web UI
-- [ ] Cost estimates per variant
-- [ ] Parallel sweeps
-- [ ] LangGraph integration
+- [x] Cost estimates per variant
+- [x] Parallel sweeps
+- [x] LangGraph integration
 
 ## License
 

{agentfork_cli-0.2.0 → agentfork_cli-0.3.0}/README.md

@@ -15,28 +15,30 @@ Everything runs locally. No account, no server, no data leaves your machine.
 A refund agent reads `"amount": 4999, "currency": "USD"` from an order API. The amount is in cents, and the agent refunds **$4,999** for a $49.99 laptop ([examples/refund_agent_cents.py](examples/refund_agent_cents.py)). Is the fix a better model or a better prompt?
 
 ```text
-$ agentfork sweep de994fe6 --step 3 --repeat 3 --expect "49.99" --model gpt-4o-mini,gpt-4o
+$ agentfork sweep 543da612 --step 3 --repeat 3 --expect "49.99" --model gpt-4o-mini,gpt-4o
+Estimated cost: about $0.0050
 
-variant             runs  pass  avg tokens  sample output
-original            1     0/1   445         I have processed a refund of $4,999.00...
-model=gpt-4o-mini   3     0/3   444         I have processed a refund of $4,999.00...
-model=gpt-4o        3     0/3   292         I apologize, it seems there was a mis...
+  variant             ok    pass   tokens   time      cost   sample output
+  original            1/1   0/1       446   1.7s   $0.0001   I have processed a refund of $4,999.00...
+  model=gpt-4o-mini   3/3   0/3       446   1.4s   $0.0001   I have processed a refund of $4,999.00...
+  model=gpt-4o        3/3   0/3       394   1.2s   $0.0015   The full amount of $4,999.00 for order A123...
 ```
 
-A bigger model doesn't fix it. One sentence in the prompt does:
+A bigger model costs 15x more per run and doesn't fix it. One sentence in the prompt does:
 
 ```text
-$ agentfork sweep de994fe6 --step 3 --repeat 3 --expect "49.99" \
+$ agentfork sweep 543da612 --step 3 --repeat 3 --expect "49.99" \
     --system "You are a support agent. Look up the order, then refund the full order amount in USD." \
     --system "... Order amounts from the API are in cents; divide by 100 before refunding."
+Estimated cost: about $0.0006
 
-variant      runs  pass  avg tokens  sample output
-original     1     0/1   445         I have processed a refund of $4,999.00...
-system #1    3     0/3   445         A full refund of $4,999.00 has ...
-system #2    3     3/3   460         I have processed a refund of $49.99...
+  variant     ok    pass   tokens   time      cost   sample output
+  original    1/1   0/1       446   1.7s   $0.0001   I have processed a refund of $4,999.00...
+  system #1   3/3   0/3       445   1.2s   $0.0001   I've successfully refunded you $4,999.00...
+  system #2   3/3   3/3       457   1.2s   $0.0001   I have processed a refund of $49.99...
 ```
 
-Steps 1 and 2 replayed from cache in every run, so only the step that mattered was paid for.
+Steps 1 and 2 replayed from cache in every run, so only the step that mattered was paid for, and the runs went 4 at a time.
 
 ## Quickstart
 
@@ -61,6 +63,33 @@ def run_agent(question): ...
 
 Run your agent as usual. Each run is saved to `.agentfork/traces.db`.
 
+### LangGraph
+
+Wrap the chat model and put `@agentfork.tool` under LangChain's `@tool`. Forks and sweeps then work on the graph like on any other agent ([examples/langgraph_refund_agent.py](examples/langgraph_refund_agent.py)):
+
+```python
+import agentfork
+from langchain_core.tools import tool
+from langchain_openai import ChatOpenAI
+from langgraph.prebuilt import create_react_agent
+
+llm = agentfork.wrap_langchain(ChatOpenAI(model="gpt-4o-mini"))   # or ChatAnthropic
+
+@tool
+@agentfork.tool
+def lookup_order(order_id: str) -> str:
+    """Look up an order by id."""
+    ...
+
+graph = create_react_agent(llm, [lookup_order])
+
+@agentfork.trace
+def run_agent(question):
+    return graph.invoke({"messages": [("user", question)]})["messages"][-1].content
+```
+
+Streaming models (`streaming=True`) and the OpenAI Responses API aren't recorded yet.
+
 ## Commands
 
 | Command | What it does |
@@ -68,7 +97,7 @@ Run your agent as usual. Each run is saved to `.agentfork/traces.db`.
 | `agentfork list` | Recent runs with steps, tokens, duration and fork lineage |
 | `agentfork show <id> [--full]` | A run as a step tree, with optional full JSON |
 | `agentfork fork <id> --step N --output/--model/--system ...` | Re-run from step N with one change; earlier steps come from cache |
-| `agentfork sweep <id> --step N --model a,b --system ... --repeat K --expect TEXT` | Try many variants of step N, K times each, and compare pass rate, tokens and time |
+| `agentfork sweep <id> --step N --model a,b --system ... --repeat K --expect TEXT [--parallel P]` | Try many variants of step N, K times each, P at a time (default 4), and compare pass rate, tokens, time and cost |
 | `agentfork diff <a> <b>` | Compare two runs step by step |
 | `agentfork save-test <id>` | Save a run as a pytest test that replays from cache in CI with no API key |
 | `agentfork ui [--port N]` | Open the local web UI to browse, fork, sweep and compare runs |
@@ -99,11 +128,19 @@ Start it from the folder you ran your agent in, since runs are saved to `.agentf
 
 ## How it works
 
-1. **Record.** `@trace`, `@tool` and `wrap_openai`/`wrap_anthropic` capture each step's inputs, outputs, tokens and latency, plus a hash of its inputs.
+1. **Record.** `@trace`, `@tool` and `wrap_openai`/`wrap_anthropic`/`wrap_langchain` capture each step's inputs, outputs, tokens and latency, plus a hash of its inputs.
 2. **Fork.** agentfork re-imports your agent and re-runs it. Steps before N are served from the cache, step N gets your change, and everything after runs live. A model change applies to every LLM call from step N on.
-3. **Sweep.** Runs fork for each variant `--repeat` times and checks each final output for your `--expect` text.
+3. **Sweep.** Runs fork for each variant `--repeat` times, `--parallel` at a time, and checks each final output for your `--expect` text. Before it starts, it estimates the cost from the tokens the original run used from step N on, priced per model.
 4. **Test.** `save-test` writes a JSON snapshot plus a pytest file. The test replays every step from cache and fails with a diff if any step's inputs, the step count, or the final output changes.
 
+## Costs
+
+Costs use list prices per model for OpenAI and Anthropic, built into [pricing.py](src/agentfork/pricing.py). Prices change, and a model without a price shows `-`. To add or correct one, put USD per 1M input and output tokens in `.agentfork/prices.json`:
+
+```json
+{"gpt-4o-mini": [0.15, 0.60], "my-finetune": [3.0, 12.0]}
+```
+
 ## Related work
 
 agentfork overlaps with record/replay tools like [agent-rewind](https://pypi.org/project/agent-rewind/), which supports many more providers, and [agentstep](https://pypi.org/project/agentstep/), a LangGraph time-travel debugger with a web UI. agentfork's focus is the step after replay: sweeping many variants of one step, with repeats and pass rates, to choose a fix with evidence.
@@ -114,9 +151,9 @@ agentfork overlaps with record/replay tools like [agent-rewind](https://pypi.org
 - [x] OpenAI and Anthropic support
 - [x] Sweep with repeats and pass rates
 - [x] Local web UI
-- [ ] Cost estimates per variant
-- [ ] Parallel sweeps
-- [ ] LangGraph integration
+- [x] Cost estimates per variant
+- [x] Parallel sweeps
+- [x] LangGraph integration
 
 ## License
 

agentfork_cli-0.3.0/examples/langgraph_refund_agent.py

@@ -0,0 +1,33 @@
+import json
+from langchain_core.tools import tool
+from langchain_openai import ChatOpenAI
+from langgraph.prebuilt import create_react_agent
+import agentfork
+
+# The refund agent from refund_agent_cents.py, built with LangGraph.
+# wrap_langchain records the model's calls; @agentfork.tool (under @tool) records the tools.
+llm = agentfork.wrap_langchain(ChatOpenAI(model="gpt-4o-mini"))
+
+SYSTEM = "You are a support agent. Look up the order, then refund the full order amount in USD."
+ORDERS = {"A123": {"order_id": "A123", "item": "laptop", "amount": 4999, "currency": "USD", "status": "delivered"}}
+
+@tool
+@agentfork.tool
+def lookup_order(order_id: str) -> str:
+    """Look up an order by id."""
+    return json.dumps(ORDERS.get(order_id, {"error": "not found"}))
+
+@tool
+@agentfork.tool
+def issue_refund(order_id: str, amount_usd: float) -> str:
+    """Refund an order. amount_usd is in US dollars."""
+    return f"Refunded ${amount_usd:,.2f} for order {order_id}"
+
+graph = create_react_agent(llm, [lookup_order, issue_refund], prompt=SYSTEM)
+
+@agentfork.trace
+def run_agent(question):
+    return graph.invoke({"messages": [("user", question)]})["messages"][-1].content
+
+if __name__ == "__main__":
+    print(run_agent("My headphones from order A123 arrived broken. Please refund me."))

{agentfork_cli-0.2.0 → agentfork_cli-0.3.0}/pyproject.toml

@@ -1,12 +1,12 @@
 [project]
 name = "agentfork-cli"
-version = "0.2.0"
+version = "0.3.0"
 description = "Fork your LLM agent at any step and sweep prompts, models or tool results to find what fixes it."
 readme = "README.md"
 requires-python = ">=3.9"
 license = "MIT"
 authors = [{ name = "Bimal Kumal" }]
-keywords = ["llm", "agents", "debugging", "replay", "testing", "openai", "anthropic"]
+keywords = ["llm", "agents", "debugging", "replay", "testing", "openai", "anthropic", "langgraph"]
 classifiers = [
     "Development Status :: 3 - Alpha",
     "Intended Audience :: Developers",
@@ -20,7 +20,7 @@ dependencies = ["typer", "rich"]
 openai = ["openai"]
 anthropic = ["anthropic"]
 ui = ["fastapi", "uvicorn"]
-dev = ["pytest", "openai", "anthropic", "fastapi", "uvicorn", "httpx"]
+dev = ["pytest", "openai", "anthropic", "fastapi", "uvicorn", "httpx", "langgraph", "langchain-openai"]
 
 [project.urls]
 Homepage = "https://github.com/bimal1023/agentfork"

agentfork_cli-0.3.0/src/agentfork/__init__.py

@@ -0,0 +1,3 @@
+from .recorder import trace, tool, wrap_anthropic, wrap_langchain, wrap_openai
+
+__all__ = ["trace", "tool", "wrap_openai", "wrap_anthropic", "wrap_langchain"]

{agentfork_cli-0.2.0 → agentfork_cli-0.3.0}/src/agentfork/cli.py

@@ -3,12 +3,13 @@ from datetime import datetime
 from typing import List, Optional
 
 import typer
+from rich import box
 from rich.console import Console
 from rich.syntax import Syntax
 from rich.table import Table
 from rich.tree import Tree
 
-from . import replay, store
+from . import pricing, replay, store
 
 app = typer.Typer(help="agentfork: record, inspect and replay LLM agent runs.")
 console = Console()
@@ -84,10 +85,13 @@ def show(run_id: str = typer.Argument(..., help="Run id or its first few charact
     color = STATUS_COLOR.get(run["status"], "white")
     fork = (f" [dim]fork of {run['parent_run_id'][:8]} at step {run['fork_step']}[/dim]"
             if run.get("parent_run_id") else "")
+    steps = store.get_steps(run["id"])
+    cost = pricing.steps_cost([s for s in steps if not s["cached"]])
+    cost = f" [dim]{pricing.fmt(cost)}[/dim]" if cost is not None else ""
     root = Tree(f"[bold]{run['name']}[/bold] {run['id'][:8]} "
-                f"[{color}]{run['status']}[/{color}] {_duration(run)}{fork}")
+                f"[{color}]{run['status']}[/{color}] {_duration(run)}{cost}{fork}")
     nodes = {}
-    for s in store.get_steps(run["id"]):
+    for s in steps:
         node = nodes.get(s["parent_id"], root).add(_label(s))
         if full:
             node.add(Syntax(json.dumps(_load(s["input_json"]), indent=2), "json", word_wrap=True))
@@ -195,6 +199,13 @@ def _make_labels(variants):
             labels[id(v)] = f"{key} #{counts[key]}"
     return labels
 
+def _clip(text, n):
+    return text if len(text) <= n else text[:n - 1] + "…"
+
+def _avg_cost(rows):
+    costs = [r["cost"] for r in rows]
+    return "-" if None in costs else pricing.fmt(sum(costs) / len(costs))
+
 @app.command()
 def sweep(run_id: str = typer.Argument(..., help="Run to sweep."),
           step: int = typer.Option(..., "--step", help="Step to vary (see `show`)."),
@@ -202,7 +213,8 @@ def sweep(run_id: str = typer.Argument(..., help="Run to sweep."),
           system: Optional[List[str]] = typer.Option(None, "--system", help="System prompt to try; repeat for more."),
           output: Optional[List[str]] = typer.Option(None, "--output", help="Tool result to try; repeat for more."),
           repeat: int = typer.Option(1, "--repeat", help="Runs per variant (LLMs vary run to run)."),
-          expect: Optional[str] = typer.Option(None, "--expect", help="Text a good final output contains.")):
+          expect: Optional[str] = typer.Option(None, "--expect", help="Text a good final output contains."),
+          parallel: int = typer.Option(4, "--parallel", help="How many runs to make at the same time.")):
     """Fork one step across several models, prompts or tool results and compare outcomes."""
     variants = [{"model": m.strip()} for m in (model or "").split(",") if m.strip()]
     variants += [{"system": s} for s in (system or [])]
@@ -213,7 +225,17 @@ def sweep(run_id: str = typer.Argument(..., help="Run to sweep."),
     run = _resolve(run_id)
     labels = _make_labels(variants)
     console.print(f"[bold]Sweeping step {step} of {run['id'][:8]}: "
-                  f"{len(variants)} variants x {repeat} = {len(variants) * repeat} runs[/bold]")
+                  f"{len(variants)} variants x {repeat} = {len(variants) * repeat} runs, "
+                  f"{min(parallel, len(variants) * repeat)} at a time[/bold]")
+    try:
+        est = replay.estimate(run, step, variants, repeat)
+    except ValueError:
+        est = None
+    if est and est["total"] is not None:
+        console.print(f"[dim]Estimated cost: about {pricing.fmt(est['total'])}[/dim]")
+    elif est and est["unpriced"]:
+        console.print(f"[dim]Estimated cost unknown: no price for {', '.join(est['unpriced'])} "
+                      f"(add it to .agentfork/prices.json)[/dim]")
 
     def progress(r):
         mark = "[green]ok[/green]" if r["status"] == "ok" else "[red]error[/red]"
@@ -221,7 +243,8 @@ def sweep(run_id: str = typer.Argument(..., help="Run to sweep."),
         console.print(f"  {mark}{verdict} {labels[id(r['variant'])]} [dim]{r['tokens']} tok[/dim]")
 
     try:
-        results = replay.sweep(run, step, variants, repeat=repeat, expect=expect, on_result=progress)
+        results = replay.sweep(run, step, variants, repeat=repeat, expect=expect,
+                               on_result=progress, parallel=parallel)
     except ValueError as e:
         console.print(f"[red]{e}[/red]")
         raise typer.Exit(1)
@@ -229,28 +252,30 @@ def sweep(run_id: str = typer.Argument(..., help="Run to sweep."),
     groups = {"original": [replay.baseline(run, step, expect)]}
     for r in results:
         groups.setdefault(labels[id(r["variant"])], []).append(r)
-    table = Table(title=f"Sweep of step {step} ({run['id'][:8]})", show_lines=True)
+    table = Table(title=f"Sweep of step {step} ({run['id'][:8]})", box=box.SIMPLE_HEAD,
+                  caption="tokens, time and cost are averages per run", caption_justify="left")
     table.add_column("variant", no_wrap=True)
-    table.add_column("runs", no_wrap=True, min_width=4)
-    table.add_column("ok", no_wrap=True, min_width=3)
+    table.add_column("ok", no_wrap=True)
     if expect:
-        table.add_column("pass", no_wrap=True, min_width=4)
-    table.add_column("avg tokens", no_wrap=True, min_width=6)
-    table.add_column("avg time", no_wrap=True, min_width=6)
-    table.add_column("sample output", max_width=36, overflow="ellipsis", no_wrap=True)
-    table.add_column("run ids", no_wrap=True)
+        table.add_column("pass", no_wrap=True)
+    table.add_column("tokens", no_wrap=True, justify="right")
+    table.add_column("time", no_wrap=True, justify="right")
+    table.add_column("cost", no_wrap=True, justify="right")
+    table.add_column("sample output", max_width=40)
+    table.add_column("runs", no_wrap=True)
     for label, rs in groups.items():
         n = len(rs)
         ids = [(r["run_id"] or "-")[:8] for r in rs]
-        row = [label, str(n), f"{sum(r['status'] == 'ok' for r in rs)}/{n}"]
+        row = [label, f"{sum(r['status'] == 'ok' for r in rs)}/{n}"]
         if expect:
             passed = sum(bool(r["passed"]) for r in rs)
             color = "green" if passed == n else ("yellow" if passed else "red")
             row.append(f"[{color}]{passed}/{n}[/{color}]")
         row += [f"{sum(r['tokens'] for r in rs) / n:.0f}",
                 f"{sum(r['latency_ms'] for r in rs) / n / 1000:.1f}s",
-                str(rs[0]["result"]),
-                ", ".join(ids[:2]) + (f" +{len(ids) - 2}" if len(ids) > 2 else "")]
+                _avg_cost(rs),
+                _clip(str(rs[0]["result"]), 60 if console.width >= 100 else 30),
+                ids[0] + (f" +{n - 1}" if n > 1 else "")]
         table.add_row(*row)
     console.print(table)
     for v in variants:

agentfork_cli-0.3.0/src/agentfork/pricing.py

@@ -0,0 +1,70 @@
+"""Model prices for cost estimates, in USD per 1M tokens: (input, output).
+
+Standard-tier list prices, checked October 2026. Prices change, so add or
+override models in .agentfork/prices.json, for example:
+
+    {"gpt-4o-mini": [0.15, 0.60], "my-finetune": [3.0, 12.0]}
+"""
+import json, re, warnings
+from . import store
+
+PRICES = {
+    # OpenAI
+    "gpt-5": (1.25, 10.00), "gpt-5-mini": (0.25, 2.00), "gpt-5-nano": (0.05, 0.40),
+    "gpt-4.1": (2.00, 8.00), "gpt-4.1-mini": (0.40, 1.60), "gpt-4.1-nano": (0.10, 0.40),
+    "gpt-4o": (2.50, 10.00), "gpt-4o-mini": (0.15, 0.60),
+    "o3": (2.00, 8.00), "o3-mini": (1.10, 4.40), "o4-mini": (1.10, 4.40),
+    # Anthropic
+    "claude-fable-5-1": (10.00, 50.00), "claude-fable-5": (10.00, 50.00),
+    "claude-opus-5-5": (4.00, 20.00), "claude-opus-5": (5.00, 25.00),
+    "claude-opus-4-8": (5.00, 25.00), "claude-opus-4-7": (5.00, 25.00),
+    "claude-opus-4-6": (5.00, 25.00), "claude-opus-4-5": (5.00, 25.00),
+    "claude-opus-4-1": (15.00, 75.00), "claude-opus-4": (15.00, 75.00),
+    "claude-sonnet-5-5": (2.00, 10.00), "claude-sonnet-5": (2.00, 10.00),
+    "claude-sonnet-4-6": (3.00, 15.00), "claude-sonnet-4-5": (3.00, 15.00),
+    "claude-sonnet-4": (3.00, 15.00), "claude-3-7-sonnet": (3.00, 15.00),
+    "claude-haiku-5-5": (0.10, 0.50), "claude-haiku-4-5": (1.00, 5.00),
+    "claude-3-5-haiku": (0.80, 4.00),
+}
+
+# Dated snapshots and aliases cost the same as the base model: gpt-4o-mini-2024-07-18, claude-3-5-haiku-latest.
+_SUFFIX = re.compile(r"-(\d{8}|\d{4}-\d{2}-\d{2}|latest)$")
+
+def _user_prices():
+    path = store.db_path().parent / "prices.json"
+    try:
+        return {k.lower(): tuple(v) for k, v in json.loads(path.read_text()).items()}
+    except FileNotFoundError:
+        return {}
+    except (ValueError, TypeError, AttributeError) as e:
+        warnings.warn(f"agentfork: ignoring {path}: {e}")
+        return {}
+
+def price(model):
+    """(input, output) USD per 1M tokens, or None if the model is unknown."""
+    if not model:
+        return None
+    table = {**PRICES, **_user_prices()}
+    name = model.lower()
+    return table.get(name) or table.get(_SUFFIX.sub("", name))
+
+def steps_cost(steps, model=None):
+    """Total USD for the LLM steps, priced at `model` if given. None if a model or token count is unknown."""
+    total = 0.0
+    for s in steps:
+        if s["type"] != "llm":
+            continue
+        p = price(model or s["name"])
+        if p is None or s["tokens_in"] is None:
+            return None
+        total += (s["tokens_in"] * p[0] + (s["tokens_out"] or 0) * p[1]) / 1e6
+    return total
+
+def unpriced(steps, model=None):
+    """Models among the LLM steps that have no price."""
+    return sorted({model or s["name"] for s in steps if s["type"] == "llm" and price(model or s["name"]) is None})
+
+def fmt(cost):
+    if cost is None:
+        return "-"
+    return f"${cost:.4f}" if cost < 0.01 else f"${cost:.2f}"

{agentfork_cli-0.2.0 → agentfork_cli-0.3.0}/src/agentfork/recorder.py

@@ -162,6 +162,8 @@ def trace(fn=None, *, name=None):
                 "parent_run_id": parent_run, "fork_step": replay.get("fork_step") if parent_run else None,
             })
             last_run_id = run_id
+            if replay:
+                replay["run_id"] = run_id
             t_run, t_seq, t_parent = _run_id.set(run_id), _seq.set([0]), _parent_step.set(None)
             status, output = "error", None
             try:
@@ -170,6 +172,8 @@ def trace(fn=None, *, name=None):
                 return output
             finally:
                 last_run_seq = _seq.get()[0]
+                if replay:
+                    replay["seq"] = last_run_seq
                 _run_id.reset(t_run); _seq.reset(t_seq); _parent_step.reset(t_parent)
                 store.update_run(run_id, ended_at=time.time(), status=status, output_json=dumps(output))
                 print(f"[agentfork] saved run {run_id[:8]} ({status})")
@@ -182,20 +186,48 @@ def tool(fn):
         return record_step("tool", fn.__name__, fn, args, kwargs)
     return wrapper
 
+class _RawResponse:
+    """Minimal stand-in for the SDKs' raw responses, for callers that use .with_raw_response (LangChain)."""
+    def __init__(self, parsed):
+        self._parsed, self.headers = parsed, {}
+    def parse(self):
+        return self._parsed
+
+class _WithRawResponse:
+    def __init__(self, create):
+        self.create = lambda *args, **kwargs: _RawResponse(create(*args, **kwargs))
+
+def _patch(resource, provider):
+    original = resource.create
+    def create(*args, **kwargs):
+        return record_step("llm", kwargs.get("model", "llm"), original, args, kwargs, provider=provider)
+    resource.create = create
+    resource.with_raw_response = _WithRawResponse(create)
+
 def wrap_openai(client):
     from openai.types.chat import ChatCompletion
     _LLM_DECODERS["openai"] = ChatCompletion.model_validate
-    original = client.chat.completions.create
-    def create(*args, **kwargs):
-        return record_step("llm", kwargs.get("model", "llm"), original, args, kwargs, provider="openai")
-    client.chat.completions.create = create
+    _patch(client.chat.completions, "openai")
     return client
 
 def wrap_anthropic(client):
     from anthropic.types import Message
     _LLM_DECODERS["anthropic"] = Message.model_validate
-    original = client.messages.create
-    def create(*args, **kwargs):
-        return record_step("llm", kwargs.get("model", "llm"), original, args, kwargs, provider="anthropic")
-    client.messages.create = create
+    _patch(client.messages, "anthropic")
     return client
+
+def wrap_langchain(llm):
+    """Record the calls a LangChain chat model makes, e.g. inside a LangGraph agent.
+
+    Works with ChatOpenAI and ChatAnthropic (non-streaming). Wrap the model before
+    passing it to the graph; models returned by bind_tools are handled too.
+    """
+    model = getattr(llm, "bound", llm)
+    if getattr(model, "root_client", None) is not None:
+        wrap_openai(model.root_client)
+        return llm
+    client = getattr(model, "_client", None)
+    if client is not None and hasattr(client, "messages"):
+        wrap_anthropic(client)
+        return llm
+    raise TypeError(f"agentfork can't record {type(model).__name__}; use ChatOpenAI or ChatAnthropic")