whetkit 0.1.0__tar.gz → 0.2.0__tar.gz

{whetkit-0.1.0 → whetkit-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: whetkit
-Version: 0.1.0
+Version: 0.2.0
 Summary: Measure and improve how well LLM agents select and use the tools exposed by an MCP server
 Keywords: mcp,model-context-protocol,llm,agents,evals,tool-use,tool-selection,benchmark
 Author: Mohammed Benlamlih
@@ -42,10 +42,17 @@ duplicates — via a **reversible overlay proxy** that never modifies your
 server. It re-runs the eval through the overlay and hands you a before/after
 report.
 
+Unlike MCP inspectors and testing frameworks, whetkit **closes the
+optimization loop**: it measures agent behavior, proposes a curated tool
+surface, applies it through a reversible proxy, and re-runs the same evals to
+quantify the improvement.
+
 ```text
-whetkit inspect  ──►  what does the agent actually see?
-whetkit run      ──►  how often does it pick the right tools?   (hit-rate)
-whetkit curate   ──►  fix the tool set, prove it helped          (before → after)
+whetkit doctor    ──►  ten-second lint of the tool surface        (no tasks, no API key)
+whetkit inspect   ──►  what does the agent actually see?
+whetkit generate  ──►  draft eval tasks from the inventory        (review, then run)
+whetkit run       ──►  how often does it pick the right tools?   (hit-rate)
+whetkit curate    ──►  fix the tool set, prove it helped          (before → after)
 ```
 
 ## Why tool curation matters
@@ -66,8 +73,8 @@ through an overlay, and show the hit-rate delta.
 uv tool install whetkit   # or: uvx whetkit / pipx install whetkit
 ```
 
-(Releases are published to PyPI from tags — see [RELEASING.md](RELEASING.md).
-Until the first release, use the from-source quickstart below.)
+(Released to PyPI from tags — see [RELEASING.md](RELEASING.md). PyPI can lag
+behind main; the quickstart below runs from source.)
 
 ## Quickstart (5 minutes)
 
@@ -109,9 +116,23 @@ This writes:
 - `.whetkit/report.json` — the same data, machine-readable
 - `.whetkit/traces.sqlite3` — full reasoning-path traces of every run
 
-Because the sample server's tasks fail mostly on tool selection, the curated
-overlay typically flips several tasks from MISS to HIT — that delta is the
-whole point.
+The sample server's failures are tool-selection failures, so weaker models
+flip several tasks from MISS to HIT through the overlay. Frontier models
+often ace even the messy baseline on a 14-tool server — there the delta
+shows up in the other columns instead: tools exposed, tokens per task, and
+extra calls. On large real-world servers (dozens of tools, near-duplicates)
+the hit-rate delta comes back.
+
+Stdio servers' own logs are hidden so they can't garble the output; set
+`WHETKIT_SERVER_LOGS=1` to see them when debugging a server that won't start.
+
+The plan is yours to edit: tweak a rename the optimizer got wrong, un-hide a
+tool, then re-score the curated view directly —
+
+```sh
+uv run whetkit run --server examples/sample-server --tasks examples/tasks \
+  --plan .whetkit/curation-plan.yaml --group curated-v2
+```
 
 **4. Use the curated view for real** — serve it to any MCP client:
 
@@ -128,7 +149,10 @@ back to the original world.
   2026-07-28-spec servers), a directory containing `server.json` or
   `server.py`, or a `.py`/`.json` path directly.
 - Write tasks in YAML — format reference in
-  [docs/task-format.md](docs/task-format.md).
+  [docs/task-format.md](docs/task-format.md) — or draft them:
+  `whetkit generate --server <your-server> --out tasks/generated.yaml`
+  writes candidate tasks from the tool inventory (validated against the
+  live tool list; review before trusting).
 - `--model` / `--judge-model` / `--optimizer-model` take
   `provider:model_id`, e.g. `anthropic:claude-sonnet-5` or `openai:gpt-5.2`.
   Keys come from `ANTHROPIC_API_KEY` / `OPENAI_API_KEY`.
@@ -137,8 +161,10 @@ back to the original world.
 
 | Command | What it does |
 |---|---|
+| `whetkit doctor` | Lint the tool surface: vague descriptions, cryptic names, near-duplicates, context bloat. `--json`; `--fail-on warn` for CI. |
 | `whetkit inspect` | Tool inventory: names, params, description tokens, schema complexity. |
-| `whetkit run` | Agentic eval loop with real tool execution; scored results + traces. |
+| `whetkit generate` | Draft eval tasks from the inventory; validated, review-before-trust YAML. |
+| `whetkit run` | Agentic eval loop with real tool execution; scored results + traces. `--plan` scores a curated view. |
 | `whetkit curate` | Baseline → LLM-proposed overlay plan → curated eval → before/after report. |
 | `whetkit report` | Rebuild the HTML/JSON report from stored traces. |
 | `whetkit overlay` | Serve the curated view as a stdio MCP server. |
@@ -163,7 +189,7 @@ More docs: [docs/task-format.md](docs/task-format.md) ·
 
 ```sh
 uv sync
-uv run pytest        # 55+ tests, no API key needed (scripted fake provider)
+uv run pytest        # full suite, no API key needed (scripted fake provider)
 uv run ruff check .
 ```
 

{whetkit-0.1.0 → whetkit-0.2.0}/README.md

@@ -11,10 +11,17 @@ duplicates — via a **reversible overlay proxy** that never modifies your
 server. It re-runs the eval through the overlay and hands you a before/after
 report.
 
+Unlike MCP inspectors and testing frameworks, whetkit **closes the
+optimization loop**: it measures agent behavior, proposes a curated tool
+surface, applies it through a reversible proxy, and re-runs the same evals to
+quantify the improvement.
+
 ```text
-whetkit inspect  ──►  what does the agent actually see?
-whetkit run      ──►  how often does it pick the right tools?   (hit-rate)
-whetkit curate   ──►  fix the tool set, prove it helped          (before → after)
+whetkit doctor    ──►  ten-second lint of the tool surface        (no tasks, no API key)
+whetkit inspect   ──►  what does the agent actually see?
+whetkit generate  ──►  draft eval tasks from the inventory        (review, then run)
+whetkit run       ──►  how often does it pick the right tools?   (hit-rate)
+whetkit curate    ──►  fix the tool set, prove it helped          (before → after)
 ```
 
 ## Why tool curation matters
@@ -35,8 +42,8 @@ through an overlay, and show the hit-rate delta.
 uv tool install whetkit   # or: uvx whetkit / pipx install whetkit
 ```
 
-(Releases are published to PyPI from tags — see [RELEASING.md](RELEASING.md).
-Until the first release, use the from-source quickstart below.)
+(Released to PyPI from tags — see [RELEASING.md](RELEASING.md). PyPI can lag
+behind main; the quickstart below runs from source.)
 
 ## Quickstart (5 minutes)
 
@@ -78,9 +85,23 @@ This writes:
 - `.whetkit/report.json` — the same data, machine-readable
 - `.whetkit/traces.sqlite3` — full reasoning-path traces of every run
 
-Because the sample server's tasks fail mostly on tool selection, the curated
-overlay typically flips several tasks from MISS to HIT — that delta is the
-whole point.
+The sample server's failures are tool-selection failures, so weaker models
+flip several tasks from MISS to HIT through the overlay. Frontier models
+often ace even the messy baseline on a 14-tool server — there the delta
+shows up in the other columns instead: tools exposed, tokens per task, and
+extra calls. On large real-world servers (dozens of tools, near-duplicates)
+the hit-rate delta comes back.
+
+Stdio servers' own logs are hidden so they can't garble the output; set
+`WHETKIT_SERVER_LOGS=1` to see them when debugging a server that won't start.
+
+The plan is yours to edit: tweak a rename the optimizer got wrong, un-hide a
+tool, then re-score the curated view directly —
+
+```sh
+uv run whetkit run --server examples/sample-server --tasks examples/tasks \
+  --plan .whetkit/curation-plan.yaml --group curated-v2
+```
 
 **4. Use the curated view for real** — serve it to any MCP client:
 
@@ -97,7 +118,10 @@ back to the original world.
   2026-07-28-spec servers), a directory containing `server.json` or
   `server.py`, or a `.py`/`.json` path directly.
 - Write tasks in YAML — format reference in
-  [docs/task-format.md](docs/task-format.md).
+  [docs/task-format.md](docs/task-format.md) — or draft them:
+  `whetkit generate --server <your-server> --out tasks/generated.yaml`
+  writes candidate tasks from the tool inventory (validated against the
+  live tool list; review before trusting).
 - `--model` / `--judge-model` / `--optimizer-model` take
   `provider:model_id`, e.g. `anthropic:claude-sonnet-5` or `openai:gpt-5.2`.
   Keys come from `ANTHROPIC_API_KEY` / `OPENAI_API_KEY`.
@@ -106,8 +130,10 @@ back to the original world.
 
 | Command | What it does |
 |---|---|
+| `whetkit doctor` | Lint the tool surface: vague descriptions, cryptic names, near-duplicates, context bloat. `--json`; `--fail-on warn` for CI. |
 | `whetkit inspect` | Tool inventory: names, params, description tokens, schema complexity. |
-| `whetkit run` | Agentic eval loop with real tool execution; scored results + traces. |
+| `whetkit generate` | Draft eval tasks from the inventory; validated, review-before-trust YAML. |
+| `whetkit run` | Agentic eval loop with real tool execution; scored results + traces. `--plan` scores a curated view. |
 | `whetkit curate` | Baseline → LLM-proposed overlay plan → curated eval → before/after report. |
 | `whetkit report` | Rebuild the HTML/JSON report from stored traces. |
 | `whetkit overlay` | Serve the curated view as a stdio MCP server. |
@@ -132,7 +158,7 @@ More docs: [docs/task-format.md](docs/task-format.md) ·
 
 ```sh
 uv sync
-uv run pytest        # 55+ tests, no API key needed (scripted fake provider)
+uv run pytest        # full suite, no API key needed (scripted fake provider)
 uv run ruff check .
 ```
 

{whetkit-0.1.0 → whetkit-0.2.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "whetkit"
-version = "0.1.0"
+version = "0.2.0"
 description = "Measure and improve how well LLM agents select and use the tools exposed by an MCP server"
 readme = "README.md"
 license = "Apache-2.0"

{whetkit-0.1.0 → whetkit-0.2.0}/src/whetkit/cli.py

@@ -2,6 +2,7 @@
 
 import asyncio
 import os
+from pathlib import Path
 from typing import Annotated
 
 import typer
@@ -39,8 +40,6 @@ def _resolve_task_servers(
 
 
 def _write_report(report, out_dir: str) -> tuple[str, str]:
-    from pathlib import Path
-
     from whetkit.report import render_html
 
     out = Path(out_dir)
@@ -52,8 +51,54 @@ def _write_report(report, out_dir: str) -> tuple[str, str]:
     return str(html_path), str(json_path)
 
 
+def _summary_payload(group_name: str, summary, task_runs: list) -> dict:
+    """One run's metrics as plain data — what --summary-json emits per run."""
+    return {
+        "group": group_name,
+        "hit_rate": summary.hit_rate,
+        "tool_hit_rate": summary.tool_hit_rate,
+        "judge_pass_rate": summary.judge_pass_rate,
+        "avg_precision": summary.avg_precision,
+        "avg_recall": summary.avg_recall,
+        "avg_extra_calls": summary.avg_extra_calls,
+        "tokens_in": sum(r.total_usage.input_tokens for r in task_runs),
+        "tokens_out": sum(r.total_usage.output_tokens for r in task_runs),
+        "latency_ms": sum(r.total_latency_ms for r in task_runs),
+        "tasks": [
+            {
+                "id": score.task_id,
+                "hit": score.hit,
+                "tool_hit": score.tool_hit,
+                "judge_passed": score.judge.passed if score.judge else None,
+                "called": score.tool_match.called,
+                "missing": score.tool_match.missing_slots,
+                "extra_calls": score.tool_match.extra_calls,
+            }
+            for score in summary.scores
+        ],
+    }
+
+
+def _version_callback(value: bool) -> None:
+    if value:
+        from importlib.metadata import version
+
+        typer.echo(f"whetkit {version('whetkit')}")
+        raise typer.Exit()
+
+
 @app.callback()
-def main() -> None:
+def main(
+    version: Annotated[
+        bool,
+        typer.Option(
+            "--version",
+            help="Show the whetkit version and exit.",
+            callback=_version_callback,
+            is_eager=True,
+        ),
+    ] = False,
+) -> None:
     """Evaluate and improve LLM agent tool selection on MCP servers."""
 
 
@@ -93,6 +138,141 @@ def inspect(
         )
 
 
+@app.command()
+def doctor(
+    server: Annotated[
+        str,
+        typer.Option("--server", help="MCP server: URL, directory, server.json, or server.py"),
+    ],
+    json_out: Annotated[
+        bool, typer.Option("--json", help="Emit findings as JSON instead of text")
+    ] = False,
+    fail_on: Annotated[
+        str,
+        typer.Option(
+            "--fail-on",
+            help="Exit non-zero when findings at this severity exist: 'error', 'warn', or 'never'",
+        ),
+    ] = "never",
+    http_mode: Annotated[HttpMode, typer.Option("--http-mode")] = HttpMode.STATEFUL,
+) -> None:
+    """Lint a server's tool surface: vague descriptions, cryptic names,
+    near-duplicates, schema and context-budget problems. No tasks or API
+    key needed."""
+    from whetkit.doctor import Severity, diagnose
+
+    if fail_on not in ("error", "warn", "never"):
+        raise typer.BadParameter("--fail-on must be 'error', 'warn', or 'never'")
+
+    spec = resolve_server_spec(server, http_mode=http_mode)
+    inventory = asyncio.run(inspect_server(spec))
+    findings = diagnose(inventory)
+
+    if json_out:
+        import json as jsonlib
+
+        typer.echo(jsonlib.dumps([f.model_dump() for f in findings], indent=2))
+    else:
+        for line in inventory.summary_lines():
+            typer.echo(line)
+        typer.echo()
+        if not findings:
+            typer.echo("No problems found — this tool surface reads clean.")
+        for finding in findings:
+            typer.echo(f"[{finding.severity.upper():<5}] {finding.check}: {finding.message}")
+        counts = {s: sum(f.severity == s for f in findings) for s in Severity}
+        if findings:
+            typer.echo(
+                f"\n{counts[Severity.ERROR]} error(s), {counts[Severity.WARN]} warning(s), "
+                f"{counts[Severity.INFO]} info — 'whetkit curate' can propose and prove fixes."
+            )
+
+    worst_hit = {
+        "error": any(f.severity == Severity.ERROR for f in findings),
+        "warn": any(f.severity in (Severity.ERROR, Severity.WARN) for f in findings),
+        "never": False,
+    }[fail_on]
+    if worst_hit:
+        raise typer.Exit(code=1)
+
+
+@app.command()
+def generate(
+    server: Annotated[
+        str,
+        typer.Option("--server", help="MCP server: URL, directory, server.json, or server.py"),
+    ],
+    out: Annotated[
+        str, typer.Option("--out", help="Where to write the drafted task YAML")
+    ] = "tasks/generated.yaml",
+    count: Annotated[int, typer.Option("--count", help="How many tasks to draft")] = 5,
+    model: Annotated[
+        str, typer.Option("--model", help="Generator model as provider:model_id")
+    ] = "anthropic:claude-sonnet-5",
+    http_mode: Annotated[HttpMode, typer.Option("--http-mode")] = HttpMode.STATEFUL,
+) -> None:
+    """Draft eval tasks from the server's tool inventory (review before
+    trusting — the header in the output file says the same)."""
+    from whetkit.generate import GeneratorConfig, generate_tasks, write_tasks_yaml
+
+    spec = resolve_server_spec(server, http_mode=http_mode)
+    inventory = asyncio.run(inspect_server(spec))
+    tasks, warnings = asyncio.run(
+        generate_tasks(inventory, server, count=count, config=GeneratorConfig(model=model))
+    )
+    for warning in warnings:
+        typer.echo(f"warning: {warning}", err=True)
+    if not tasks:
+        typer.echo("no valid tasks were drafted — try again or write them by hand", err=True)
+        raise typer.Exit(code=1)
+
+    Path(out).parent.mkdir(parents=True, exist_ok=True)
+    write_tasks_yaml(tasks, out)
+    typer.echo(f"drafted {len(tasks)} task(s) to {out} — review them, then:")
+    typer.echo(f"  whetkit run --server {server} --tasks {out}")
+
+
+@app.command()
+def export(
+    plan: Annotated[
+        str, typer.Option("--plan", help="Curation plan YAML to export")
+    ] = ".whetkit/curation-plan.yaml",
+    to: Annotated[
+        str,
+        typer.Option(
+            "--to",
+            help=(
+                "'markdown' — a fix report ready for an upstream issue/PR; "
+                "'json' — a neutral override list for gateways and scripts"
+            ),
+        ),
+    ] = "markdown",
+    out: Annotated[str | None, typer.Option("--out", help="Write here instead of stdout")] = None,
+) -> None:
+    """Export a curation plan as a shareable fix report or neutral JSON."""
+    from whetkit.curation import load_plan
+    from whetkit.curation.export import plan_to_json, plan_to_markdown
+
+    if to not in ("markdown", "json"):
+        raise typer.BadParameter("--to must be 'markdown' or 'json'")
+    if not Path(plan).is_file():
+        raise typer.BadParameter(
+            f"no curation plan at {plan} — run 'whetkit curate' first, or pass --plan"
+        )
+    curation_plan = load_plan(plan)
+    if not curation_plan.overrides:
+        typer.echo("plan has no overrides — nothing to export", err=True)
+        raise typer.Exit(code=1)
+
+    rendered = (plan_to_markdown if to == "markdown" else plan_to_json)(curation_plan)
+    if out:
+        Path(out).parent.mkdir(parents=True, exist_ok=True)
+        Path(out).write_text(rendered)
+        typer.echo(f"wrote {out}")
+    else:
+        typer.echo(rendered, nl=False)
+
+
 @app.command()
 def run(
     tasks: Annotated[
@@ -118,56 +298,109 @@ def run(
     group: Annotated[
         str, typer.Option("--group", help="Label for this batch of runs in the trace store")
     ] = "baseline",
+    plan: Annotated[
+        str | None,
+        typer.Option(
+            "--plan",
+            help=(
+                "Eval the curated view: apply this curation plan as an overlay "
+                "and score against origin tool names (for hand-tuned plans)"
+            ),
+        ),
+    ] = None,
     match_mode: Annotated[
         str, typer.Option("--match-mode", help="Tool matching: 'order_tolerant' or 'exact'")
     ] = "order_tolerant",
     max_turns: Annotated[int, typer.Option("--max-turns")] = 10,
+    max_tokens: Annotated[
+        int,
+        typer.Option(
+            "--max-tokens",
+            help="Completion-token budget per model turn (raise for reasoning models)",
+        ),
+    ] = 1024,
+    runs: Annotated[
+        int,
+        typer.Option(
+            "--runs",
+            help=(
+                "Repeat the whole task set N times and report mean plus range "
+                "— single runs are noise. Trace groups get a -1..-N suffix."
+            ),
+        ),
+    ] = 1,
     store: Annotated[
         str | None,
         typer.Option("--store", help="Trace SQLite path (default ./.whetkit/traces.sqlite3)"),
     ] = None,
     jsonl: Annotated[
-        str | None, typer.Option("--jsonl", help="Also write traces to this JSONL file")
+        str | None,
+        typer.Option(
+            "--jsonl",
+            help="Also write traces to this JSONL file (suffixed per run when --runs > 1)",
+        ),
+    ] = None,
+    summary_json: Annotated[
+        str | None,
+        typer.Option(
+            "--summary-json",
+            help="Write a machine-readable summary (metrics + per-task outcomes) to this path",
+        ),
     ] = None,
     http_mode: Annotated[HttpMode, typer.Option("--http-mode")] = HttpMode.STATEFUL,
 ) -> None:
     """Run eval tasks against an MCP server and print scored results."""
     from whetkit.datasets import load_tasks
     from whetkit.runner import RunConfig, run_task
-    from whetkit.scoring import JudgeCache, JudgeConfig, MatchMode, score_runs
+    from whetkit.scoring import JudgeCache, JudgeConfig, MatchMode, MultiRunSummary, score_runs
     from whetkit.tracing import TraceStore, default_store_path, write_jsonl
 
+    if runs < 1:
+        raise typer.BadParameter("--runs must be at least 1")
+
     task_list = load_tasks(tasks)
     servers = _resolve_task_servers(task_list, server, http_mode)
-    config = RunConfig(model=model, max_turns=max_turns)
+    config = RunConfig(model=model, max_turns=max_turns, max_tokens=max_tokens)
     use_judge = _judge_enabled(judge, judge_model)
     store_path = store or str(default_store_path())
 
-    async def _run() -> None:
-        runs = []
+    from whetkit.mcp import MCPClient
+
+    client_factory = MCPClient
+    name_map: dict[str, str] | None = None
+    if plan is not None:
+        from functools import partial
+
+        from whetkit.curation import CuratedMCPClient, load_plan
+
+        if not Path(plan).is_file():
+            raise typer.BadParameter(f"no curation plan at {plan}")
+        curation_plan = load_plan(plan)
+        client_factory = partial(CuratedMCPClient, plan=curation_plan)
+        name_map = curation_plan.rename_map()
+
+    async def _run_once(group_name: str, cache: JudgeCache):
+        task_runs = []
         for task in task_list:
             typer.echo(f"running {task.id} ...", err=True)
-            runs.append(await run_task(task, servers[task.server], config))
+            task_runs.append(
+                await run_task(task, servers[task.server], config, client_factory=client_factory)
+            )
 
         with TraceStore(store_path) as trace_store:
-            trace_store.save_runs(runs, run_group=group)
-        if jsonl:
-            write_jsonl(runs, jsonl)
+            trace_store.save_runs(task_runs, run_group=group_name)
 
-        cache = JudgeCache(default_store_path().parent / "judge_cache.sqlite3")
-        try:
-            summary = await score_runs(
-                task_list,
-                runs,
-                mode=MatchMode(match_mode),
-                judge_config=JudgeConfig(model=judge_model),
-                judge_cache=cache,
-                use_judge=use_judge,
-            )
-        finally:
-            cache.close()
+        summary = await score_runs(
+            task_list,
+            task_runs,
+            mode=MatchMode(match_mode),
+            judge_config=JudgeConfig(model=judge_model),
+            judge_cache=cache,
+            use_judge=use_judge,
+            name_map=name_map,
+        )
 
-        typer.echo(f"\nResults (group '{group}', model {model}):")
+        typer.echo(f"\nResults (group '{group_name}', model {model}):")
         for score in summary.scores:
             mark = "PASS" if score.hit else "MISS"
             tools = " -> ".join(score.tool_match.called) or "(no tool calls)"
@@ -180,11 +413,62 @@ def run(
         typer.echo("")
         for line in summary.summary_lines():
             typer.echo(line)
-        if not use_judge:
+        tokens_in = sum(r.total_usage.input_tokens for r in task_runs)
+        tokens_out = sum(r.total_usage.output_tokens for r in task_runs)
+        typer.echo(
+            f"Tokens in/out: {tokens_in}/{tokens_out}   "
+            f"Total latency: {sum(r.total_latency_ms for r in task_runs) / 1000:.1f}s"
+        )
+        return task_runs, summary
+
+    async def _run() -> None:
+        summaries = []
+        payloads = []
+        cache = JudgeCache(default_store_path().parent / "judge_cache.sqlite3")
+        try:
+            for run_index in range(1, runs + 1):
+                group_name = group if runs == 1 else f"{group}-{run_index}"
+                task_runs, summary = await _run_once(group_name, cache)
+                summaries.append(summary)
+                payloads.append(_summary_payload(group_name, summary, task_runs))
+                if jsonl:
+                    write_jsonl(task_runs, jsonl if runs == 1 else f"{jsonl}.{run_index}")
+        finally:
+            cache.close()
+
+        multi = MultiRunSummary(summaries=summaries)
+        if runs > 1:
+            typer.echo(f"\n== across {runs} runs ==")
+            for line in multi.summary_lines():
+                typer.echo(line)
+        if judge == "auto" and not use_judge:
             typer.echo(
                 "(LLM judge skipped: no API key found — set ANTHROPIC_API_KEY or pass --judge on)"
             )
-        typer.echo(f"Traces saved to {store_path} (group '{group}')")
+        suffix = f" (groups '{group}-1'..'-{runs}')" if runs > 1 else f" (group '{group}')"
+        typer.echo(f"Traces saved to {store_path}{suffix}")
+
+        if summary_json:
+            import json as jsonlib
+
+            document = {
+                "model": model,
+                "judge_model": judge_model if use_judge else None,
+                "match_mode": match_mode,
+                "plan": plan,
+                "runs": payloads,
+            }
+            if runs > 1:
+                document["aggregate"] = {
+                    "n": multi.n,
+                    "hit_rate_mean": sum(s.hit_rate for s in summaries) / len(summaries),
+                    "hit_rate_min": min(s.hit_rate for s in summaries),
+                    "hit_rate_max": max(s.hit_rate for s in summaries),
+                    "flaky_tasks": multi.flaky_tasks(),
+                }
+            Path(summary_json).parent.mkdir(parents=True, exist_ok=True)
+            Path(summary_json).write_text(jsonlib.dumps(document, indent=2) + "\n")
+            typer.echo(f"Summary JSON: {summary_json}")
 
     asyncio.run(_run())
 
@@ -214,6 +498,13 @@ def curate(
     ] = ".whetkit",
     match_mode: Annotated[str, typer.Option("--match-mode")] = "order_tolerant",
     max_turns: Annotated[int, typer.Option("--max-turns")] = 10,
+    max_tokens: Annotated[
+        int,
+        typer.Option(
+            "--max-tokens",
+            help="Completion-token budget per model turn (raise for reasoning models)",
+        ),
+    ] = 1024,
     store: Annotated[str | None, typer.Option("--store")] = None,
     http_mode: Annotated[HttpMode, typer.Option("--http-mode")] = HttpMode.STATEFUL,
 ) -> None:
@@ -229,7 +520,7 @@ def curate(
 
     task_list = load_tasks(tasks)
     servers = _resolve_task_servers(task_list, server, http_mode)
-    config = RunConfig(model=model, max_turns=max_turns)
+    config = RunConfig(model=model, max_turns=max_turns, max_tokens=max_tokens)
     use_judge = _judge_enabled(judge, judge_model)
     judge_config = JudgeConfig(model=judge_model)
     mode = MatchMode(match_mode)
@@ -285,6 +576,7 @@ def curate(
                 judge_config=judge_config,
                 judge_cache=cache,
                 use_judge=use_judge,
+                name_map=plan.rename_map(),
             )
         finally:
             cache.close()
@@ -326,6 +618,12 @@ def curate(
             f"Tool hit-rate: {baseline.tool_hit_rate:.0%} -> {curated.tool_hit_rate:.0%}   "
             f"Precision: {baseline.avg_precision:.0%} -> {curated.avg_precision:.0%}"
         )
+        tok_before = (report.before.input_tokens + report.before.output_tokens) // len(task_list)
+        tok_after = (report.after.input_tokens + report.after.output_tokens) // len(task_list)
+        typer.echo(
+            f"Tools: {report.tools_before} -> {report.tools_after}   "
+            f"Tokens/task: {tok_before} -> {tok_after}"
+        )
         typer.echo(f"Traces saved to {store_path} (groups 'baseline', 'curated')")
         typer.echo(f"Report: {html_path} (machine-readable: {json_path})")
 
@@ -357,6 +655,10 @@ def report(
     from whetkit.tracing import TraceStore, default_store_path
 
     task_list = load_tasks(tasks)
+    if not Path(plan).is_file():
+        raise typer.BadParameter(
+            f"no curation plan at {plan} — run 'whetkit curate' first, or pass --plan"
+        )
     curation_plan = load_plan(plan)
     store_path = store or str(default_store_path())
     use_judge = _judge_enabled(judge, judge_model)
@@ -390,6 +692,7 @@ def report(
                 judge_config=judge_config,
                 judge_cache=cache,
                 use_judge=use_judge,
+                name_map=curation_plan.rename_map(),
             )
         finally:
             cache.close()