agent-readable 0.2.1__tar.gz → 0.3.0__tar.gz

{agent_readable-0.2.1 → agent_readable-0.3.0}/.github/workflows/ci.yml

@@ -24,6 +24,9 @@ jobs:
       - name: Ruff format check
         run: uv run ruff format --check
 
+      - name: Mypy
+        run: uv run mypy
+
   test:
     runs-on: ubuntu-latest
     strategy:

{agent_readable-0.2.1 → agent_readable-0.3.0}/.gitignore

@@ -43,3 +43,6 @@ hatch.toml
 
 # No external dependencies for this project, and uv.lock is only needed for local dev.
 uv.lock
+
+# Local scratch files (audits, drafts, planning docs)
+.localonly/

{agent_readable-0.2.1 → agent_readable-0.3.0}/CHANGELOG.md

@@ -7,6 +7,54 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.3.0] - 2026-08-23
+
+### Added
+
+- Console script `agent-readable`, so one-off use is
+  `uvx agent-readable sqlite3:Connection` instead of
+  `uvx --from agent-readable python -m agent_readable sqlite3:Connection`.
+  `python -m agent_readable` keeps working.
+- `--version` flag for the CLI.
+- Public class attributes and module constants are now listed in the Public API
+  section with their current value: `repr` for exact primitive types (never
+  executing a custom `__repr__`), the type name otherwise.
+- Enum members are listed (e.g. ``- `RED` member: 1``), and the misleading
+  `EnumMeta.__call__` constructor signature is no longer shown for enum
+  classes.
+- Module docs now include C builtins (`inspect.isroutine` instead of
+  `inspect.isfunction`), so `agent_help(math)` lists `sin` alongside
+  pure-Python functions, and module-level constants such as `math.pi` are no
+  longer dropped by the origin heuristic.
+- A constructor that introspection renders as the placeholder `(*args,
+  **kwargs)` — e.g. masked by a metaclass `__call__` — now falls back to the
+  class's `__init__`/`__new__` signature.
+- Mypy type-checking job in CI.
+- Benchmark harness (`scripts/benchmark.py`) and methodology
+  (`docs/benchmark.md`) for measuring whether `agent_help()` context reduces
+  hallucinated-API failures versus a baseline prompt. Status: not yet run;
+  no numbers are claimed until a real run is recorded.
+
+### Changed
+
+- **Breaking:** `__agent_notes__()` sections are now always appended, including
+  after a custom `__agent_help__()`'s output. Previously the notes were
+  silently dropped when a custom `__agent_help__` was defined and a
+  `UserWarning` was emitted; defining both is now a supported combination and
+  the warning is gone. For full verbatim control of the entire output, do not
+  define `__agent_notes__()` anywhere in the MRO.
+- The CLI prints a one-line `error: ...` message to stderr and exits with
+  status 2 for targets that cannot be imported or resolved, instead of raising
+  a traceback.
+
+### Fixed
+
+- A raising `__agent_notes__()` no longer breaks `agent_help()` for the class
+  and all of its subclasses; the broken notes are skipped, mirroring the
+  existing `__agent_help__()` fallback.
+- `functools.cached_property` members are no longer silently dropped from the
+  Public API; they render as properties with the wrapped function's docstring.
+
 ## [0.2.1] - 2026-07-04
 
 ### Documentation
@@ -86,7 +134,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   `AgentReadableMixin`, `__agent_notes__` accumulation across the MRO, module
   support, and the `python -m agent_readable` CLI.
 
-[Unreleased]: https://github.com/zydo/agent-readable/compare/v0.2.1...HEAD
+[Unreleased]: https://github.com/zydo/agent-readable/compare/v0.3.0...HEAD
+[0.3.0]: https://github.com/zydo/agent-readable/compare/v0.2.1...v0.3.0
 [0.2.1]: https://github.com/zydo/agent-readable/compare/v0.2.0...v0.2.1
 [0.1.2]: https://github.com/zydo/agent-readable/compare/v0.1.1...v0.1.2
 [0.1.1]: https://github.com/zydo/agent-readable/compare/v0.1.0...v0.1.1

{agent_readable-0.2.1 → agent_readable-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: agent-readable
-Version: 0.2.1
+Version: 0.3.0
 Summary: A lightweight Python protocol for agent-oriented documentation
 Project-URL: Repository, https://github.com/zydo/agent-readable
 Project-URL: Issues, https://github.com/zydo/agent-readable/issues
@@ -40,7 +40,7 @@ npx skills add zydo/skills --skill agent-readable
 <!-- markdownlint-disable MD033 -->
 <p align="center">
   <strong><code>logging.Logger</code> compared with <code>agent_help()</code> and <code>help()</code></strong><br>
-  <img src="docs/agent_help_vs_help.gif" alt="agent_help vs help">
+  <img src="https://raw.githubusercontent.com/zydo/agent-readable/main/docs/agent_help_vs_help.gif" alt="agent_help vs help">
 </p>
 <!-- markdownlint-enable MD033 -->
 
@@ -86,6 +86,7 @@ class Sensor:
 - [Examples](docs/examples.md): wrapping existing classes, inherited notes, duck typing, plain classes, modules, functions, and methods.
 - [Authoring guide](docs/authoring.md): `__agent_help__`, `__agent_notes__`, class docstring hints, freshness guidance, and API reference.
 - [FAQ](docs/faq.md): common questions about agent skills, docstrings, `AGENTS.md`, third-party libraries, and constrained decoding.
+- [Benchmark](docs/benchmark.md): methodology and harness measuring whether `agent_help()` context reduces hallucinated-API failures.
 
 ## Other Languages
 

{agent_readable-0.2.1 → agent_readable-0.3.0}/README.md

@@ -13,7 +13,7 @@ npx skills add zydo/skills --skill agent-readable
 <!-- markdownlint-disable MD033 -->
 <p align="center">
   <strong><code>logging.Logger</code> compared with <code>agent_help()</code> and <code>help()</code></strong><br>
-  <img src="docs/agent_help_vs_help.gif" alt="agent_help vs help">
+  <img src="https://raw.githubusercontent.com/zydo/agent-readable/main/docs/agent_help_vs_help.gif" alt="agent_help vs help">
 </p>
 <!-- markdownlint-enable MD033 -->
 
@@ -59,6 +59,7 @@ class Sensor:
 - [Examples](docs/examples.md): wrapping existing classes, inherited notes, duck typing, plain classes, modules, functions, and methods.
 - [Authoring guide](docs/authoring.md): `__agent_help__`, `__agent_notes__`, class docstring hints, freshness guidance, and API reference.
 - [FAQ](docs/faq.md): common questions about agent skills, docstrings, `AGENTS.md`, third-party libraries, and constrained decoding.
+- [Benchmark](docs/benchmark.md): methodology and harness measuring whether `agent_help()` context reduces hallucinated-API failures.
 
 ## Other Languages
 

{agent_readable-0.2.1 → agent_readable-0.3.0}/docs/authoring.md

@@ -28,13 +28,13 @@ The two dunders intentionally encode different composition rules:
 
 | Aspect          | `__agent_help__()`                         | `__agent_notes__()`                                                           |
 | --------------- | ------------------------------------------ | ----------------------------------------------------------------------------- |
-| Semantics       | Replacement: returned string is the output | Additive: appended to auto-generated docs                                     |
+| Semantics       | Replaces the auto-generated base document  | Additive: appended after the auto-doc or custom `__agent_help__()` output     |
 | Composition     | Single class wins, closest in MRO          | Accumulated across the MRO; leaf class wins on conflict                       |
-| When to use     | Total control over rendered text           | Auto-doc plus extra do/don't rules                                            |
-| Skipped when    | Always called if defined                   | Silently dropped, with `UserWarning`, when custom `__agent_help__` is present |
+| When to use     | Custom control over the base document      | Extra do/don't rules on top of any base                                       |
+| Skipped when    | Always called if defined                   | Never skipped when defined; a notes method that raises is skipped             |
 | Mixin required? | No                                         | No                                                                            |
 
-When a class defines both `__agent_help__()` and `__agent_notes__()`, the notes are silently dropped because `__agent_help__()` owns the output and the auto-doc plus notes path never runs. A `UserWarning` is emitted, but warnings are easy to miss in agent shells, CI logs, and notebooks. Treat "both defined" as a review error. Fix it by folding the notes into `__agent_help__()`, or by dropping custom `__agent_help__()` and letting the auto-doc plus notes path run.
+When a class defines both `__agent_help__()` and `__agent_notes__()`, the custom help replaces the auto-generated base document and the notes are appended after it — the same additive behavior as the auto-doc path, so no combination of the two dunders silently drops notes. If you need full verbatim control of the entire output, do not define `__agent_notes__()` anywhere in the class hierarchy.
 
 ## Class docstring hints
 
@@ -58,9 +58,9 @@ This way, even agents that only see the source or call `help()` are reminded to
 
 Render agent-oriented Markdown for a class, module, function, or method.
 
-For classes, the output includes purpose, constructor, public API, default usage rules, and accumulated `__agent_notes__()` sections when present. For modules, it includes purpose, public functions, and public classes. For functions and methods, it includes signature, docstring, and default usage rules.
+For classes, the output includes purpose, constructor, public API (methods, properties, cached properties, constants, and enum members), default usage rules, and accumulated `__agent_notes__()` sections when present. For modules, it includes purpose, public functions (including C builtins), public classes, and constants. For functions and methods, it includes signature, docstring, and default usage rules.
 
-For classes and instances, if `__agent_help__()` is defined through the mixin or duck typing, it is called and its return value is used verbatim. Duck-typed implementations are responsible for their own formatting, and notes are not auto-appended. If such a class also defines `__agent_notes__()`, a `UserWarning` is emitted because those notes are silently dropped. Fold them into `__agent_help__()`, or drop the custom `__agent_help__()` to use the auto-doc path. If `__agent_help__()` raises, `agent_help()` falls back to the auto-generated path, which does include notes.
+For classes and instances, if `__agent_help__()` is defined through the mixin or duck typing, it is called and its return value is used as the base document. Duck-typed implementations replace the auto-generated base; `__agent_notes__()` sections from the MRO are still appended after it (the mixin default already embeds notes, so its output is used as-is). If `__agent_help__()` raises, `agent_help()` falls back to the auto-generated path, which does include notes. A `__agent_notes__()` that raises is skipped rather than fatal, so one broken notes method cannot take down help for a class and its subclasses.
 
 For modules, if the module defines a `__agent_help__` attribute, either callable or string, it is used. Otherwise, auto-generated docs are produced from the module docstring and its public functions and classes. When the module defines `__all__`, that list is the authoritative public API, so re-exported symbols are included. Otherwise public members are discovered by introspection, skipping private names and anything defined outside the module.
 

agent_readable-0.3.0/docs/benchmark.md

@@ -0,0 +1,79 @@
+# Benchmark: does `agent_help()` context reduce API hallucination?
+
+**Status: not yet run.** This page documents the methodology and the harness
+(`scripts/benchmark.py`). No numbers are published until a real run has been
+executed and recorded here — claims in the README about hallucination and
+token efficiency remain unevidenced until then.
+
+## The question
+
+When a coding agent writes Python against an API it has not seen (a new
+library, a recent release, a project-private module), does injecting
+`agent_help(target)` output into the prompt reduce failures compared with the
+same prompt without it?
+
+The failure modes scored are the two documented in [Why it matters](why.md):
+
+- **What exists** — hallucinated attributes and methods, measured as
+  `AttributeError` at runtime.
+- **How to use it** — wrong signatures or call shapes, measured as `TypeError`
+  at runtime.
+
+## Method
+
+For each task (one target library plus a natural-language coding prompt) and
+each condition, the harness asks the model for a single runnable script and
+executes it against the installed library:
+
+| Condition | Prompt |
+| --- | --- |
+| `baseline` | The task prompt only. |
+| `agent_help` | The task prompt plus `agent_help(target)` output labeled as reference documentation for the installed version. |
+
+Every generated script runs in a fresh subprocess against the same
+environment. Outcomes:
+
+| Outcome | Meaning |
+| --- | --- |
+| `ok` | Script exits 0. |
+| `attributeerror` | Runtime `AttributeError` — most directly indicates a hallucinated API. |
+| `typeerror` | Runtime `TypeError` — wrong signature or call shape. |
+| `nameerror`, `importerror` | Reference or import failures. |
+| `other_error`, `timeout`, `no_code`, `api_error` | Everything else; reported but not counted as API hallucination. |
+
+Headline metric: the `ok` rate per condition, plus the split between
+`attributeerror` and `typeerror`. Results are recorded below with the date,
+model, sample count, and library versions.
+
+## Running it
+
+The harness calls the Anthropic API (needs `ANTHROPIC_API_KEY` or an
+`ant auth login` profile) and requires the target libraries to be installed.
+It is deliberately not part of CI.
+
+```bash
+uv run --with anthropic --with icalendar --with feedparser \
+    python scripts/benchmark.py --samples 5 --out .localonly/benchmark-results.json
+```
+
+Custom task sets are JSON files of `{module, target, task}` entries passed via
+`--tasks`. Choose targets the model plausibly has weak training coverage of:
+small libraries, recent releases, or project-private code — for well-known
+stable APIs the model may succeed in both conditions and the benchmark
+measures nothing.
+
+## Threats to validity
+
+- **Target familiarity.** A model that already knows a library reduces the
+  gap; a model that misremembers a *changed* API may fail both conditions.
+  Unfamiliar targets are the population of interest.
+- **Task wording.** The injected documentation is the only intended
+  difference; prompts are otherwise identical.
+- **Execution-only scoring.** A script can exit 0 and still be semantically
+  wrong; `ok` is an upper bound on correctness, not a proof of it.
+- **Sample size.** Per-task differences at small sample counts are noise;
+  read the aggregate.
+
+## Results
+
+_None yet._

{agent_readable-0.2.1 → agent_readable-0.3.0}/docs/examples.md

@@ -154,11 +154,11 @@ class RateLimiter:
 print(agent_help(RateLimiter))
 ```
 
-Use `__agent_help__()` when you need total control over the rendered output. Use `__agent_notes__()` when you want auto-generated API docs plus your extra rules.
+Use `__agent_help__()` when the auto-generated base document is not what you want and you prefer to write the base by hand. Use `__agent_notes__()` when the auto-generated API docs are fine and you just want extra rules on top. `__agent_notes__()` sections defined anywhere in the MRO are appended after custom `__agent_help__()` output too, so mixed setups do not lose notes.
 
 ## Example 4: Any class, no setup required
 
-Even without the mixin or duck typing, `agent_help()` generates structured Markdown from introspection: a curated public API list with current signatures, free of inherited dunders and MRO clutter. Full example: [`examples/any_class.py`](../examples/any_class.py).
+Even without the mixin or duck typing, `agent_help()` generates structured Markdown from introspection: a curated public API list with current signatures, free of inherited dunders and MRO clutter. Methods, properties, cached properties, public constants (with their values), and enum members are all listed. Full example: [`examples/any_class.py`](../examples/any_class.py).
 
 ```python
 import logging

{agent_readable-0.2.1 → agent_readable-0.3.0}/docs/getting-started.md

@@ -23,22 +23,22 @@ The examples below use `sqlite3:Connection` as the target; replace it with any i
 With `uvx`:
 
 ```bash
-uvx --from agent-readable python -m agent_readable sqlite3:Connection
+uvx --from agent-readable agent-readable sqlite3:Connection
 ```
 
 With `uv tool run`:
 
 ```bash
-uv tool run --from agent-readable python -m agent_readable sqlite3:Connection
+uv tool run --from agent-readable agent-readable sqlite3:Connection
 ```
 
 With `pipx`:
 
 ```bash
-pipx run --spec agent-readable python -m agent_readable sqlite3:Connection
+pipx run --spec agent-readable agent-readable sqlite3:Connection
 ```
 
-`uvx` is shorthand for `uv tool run`. These commands are useful for standard-library targets and packages available inside the temporary environment. For your own project classes, run from an environment where that project is importable.
+`uvx` is shorthand for `uv tool run`, and `python -m agent_readable` works identically wherever the package is installed. These commands are useful for standard-library targets and packages available inside the temporary environment. For your own project classes, run from an environment where that project is importable.
 
 To let your coding agent automatically call `agent_help()` before using an unfamiliar API, install the companion skill:
 
@@ -123,25 +123,30 @@ print(agent_help(Sensor))
 
 ## CLI
 
+Wherever the package is installed, both `agent-readable` and `python -m agent_readable` work:
+
 ```bash
 # Any stdlib class
-python -m agent_readable sqlite3:Connection
+agent-readable sqlite3:Connection
 
 # A class in your own package
-python -m agent_readable my_package.temperature:CalibratedSensor
+agent-readable my_package.temperature:CalibratedSensor
 
 # The library itself
-python -m agent_readable agent_readable:AgentReadableMixin
+agent-readable agent_readable:AgentReadableMixin
 
 # Any module
-python -m agent_readable pathlib
+agent-readable pathlib
 
 # A function or method
-python -m agent_readable json:dumps
-python -m agent_readable pathlib:Path.read_text
+agent-readable json:dumps
+agent-readable pathlib:Path.read_text
+
+# Installed version
+agent-readable --version
 ```
 
-The CLI writes agent-oriented documentation for the target to stdout.
+The CLI writes agent-oriented documentation for the target to stdout. A target that cannot be imported or resolved prints a one-line error to stderr and exits with status 2.
 
 ## Other Languages
 

{agent_readable-0.2.1 → agent_readable-0.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "agent-readable"
-version = "0.2.1"
+version = "0.3.0"
 description = "A lightweight Python protocol for agent-oriented documentation"
 readme = "README.md"
 license = "MIT"
@@ -47,6 +47,9 @@ Repository = "https://github.com/zydo/agent-readable"
 Issues = "https://github.com/zydo/agent-readable/issues"
 Changelog = "https://github.com/zydo/agent-readable/blob/main/CHANGELOG.md"
 
+[project.scripts]
+agent-readable = "agent_readable.__main__:main"
+
 [tool.hatch.build.targets.wheel]
 packages = ["src/agent_readable"]
 
@@ -61,9 +64,17 @@ target-version = "py310"
 select = ["E", "F", "I", "UP", "B", "RUF"]
 external = ["S"]
 
+[tool.mypy]
+files = ["src", "tests"]
+python_version = "3.10"
+check_untyped_defs = true
+warn_redundant_casts = true
+no_implicit_optional = true
+
 [dependency-groups]
 dev = [
     "pytest>=8.0",
     "pytest-cov>=7.1.0",
     "ruff>=0.15.15",
+    "mypy>=1.19.0",
 ]

agent_readable-0.3.0/scripts/benchmark.py

@@ -0,0 +1,243 @@
+"""Benchmark: does `agent_help()` context reduce API-hallucination failures?
+
+Compares two conditions on identical code-generation tasks against unfamiliar
+libraries:
+
+- ``baseline``: a natural-language task prompt only.
+- ``agent_help``: the same prompt with `agent_help(target)` output injected as
+  reference documentation.
+
+Each generated script is executed against the installed library and scored by
+outcome: clean exit, AttributeError (hallucinated API), TypeError (wrong
+signature/usage), other error, timeout, or no code returned.
+
+Not wired into CI: it calls a paid API and needs the target libraries
+installed. See docs/benchmark.md for methodology and current status.
+
+Usage:
+
+    uv run --with anthropic --with <target-libraries> \
+        python scripts/benchmark.py [--tasks tasks.json] \
+        [--samples 3] [--out results.json]
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import subprocess
+import sys
+import tempfile
+from dataclasses import asdict, dataclass
+from pathlib import Path
+from typing import Any
+
+import anthropic
+
+from agent_readable import agent_help
+
+MODEL = "claude-opus-5"
+RUN_TIMEOUT_S = 30
+
+BASELINE_PROMPT = """You are writing Python code using the `{module}` library.
+
+Task: {task}
+
+Reply with exactly one Python code block containing a complete, runnable
+script. No explanations, no example usage outside the block.
+"""
+
+TREATMENT_PROMPT = """You are writing Python code using the `{module}` library.
+
+Reference documentation for `{target}`, generated from the installed version:
+
+{doc}
+
+Task: {task}
+
+Reply with exactly one Python code block containing a complete, runnable
+script. No explanations, no example usage outside the block.
+"""
+
+
+@dataclass(frozen=True)
+class Task:
+    """One code-generation task against one importable target."""
+
+    module: str
+    target: str
+    task: str
+
+
+@dataclass
+class SampleResult:
+    condition: str
+    module: str
+    sample: int
+    outcome: str
+    detail: str = ""
+
+
+# Tasks deliberately favor small, unfamiliar libraries so the model cannot
+# answer from training data. Add entries via --tasks <file> with the same
+# shape; the listed libraries must be installed in the run environment.
+DEFAULT_TASKS = (
+    Task(
+        module="icalendar",
+        target="icalendar:Calendar",
+        task="Parse the ICS string 'BEGIN:VCALENDAR\\r\\nEND:VCALENDAR\\r\\n' "
+        "and print the calendar's PRODID property (or 'none' if absent).",
+    ),
+    Task(
+        module="feedparser",
+        target="feedparser",
+        task='Parse the Atom feed string \'<feed xmlns="http://www.w3.org/2005/Atom">'
+        "<title>t</title></feed>' and print the feed title.",
+    ),
+)
+
+
+def generate(client: anthropic.Anthropic, prompt: str) -> str:
+    """Call the model and return its text, or raise RuntimeError on refusal."""
+    response = client.messages.create(
+        model=MODEL,
+        max_tokens=4096,
+        messages=[{"role": "user", "content": prompt}],
+    )
+    if response.stop_reason == "refusal":
+        raise RuntimeError(f"model refused ({response.stop_details})")
+    return "".join(block.text for block in response.content if block.type == "text")
+
+
+def extract_code(response_text: str) -> str | None:
+    """Return the first fenced code block, or None when the reply has none."""
+    for match in re.finditer(r"```(?:python)?\s*\n(.*?)```", response_text, re.DOTALL):
+        return match.group(1)
+    return None
+
+
+def classify(code: str) -> SampleResult:
+    """Execute one generated script and classify the outcome."""
+    with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as f:  # noqa: SIM115 - executed before deletion below
+        f.write(code)
+        path = f.name
+    try:
+        proc = subprocess.run(
+            [sys.executable, path],
+            capture_output=True,
+            text=True,
+            timeout=RUN_TIMEOUT_S,
+        )
+    except subprocess.TimeoutExpired:
+        return SampleResult("", "", 0, "timeout")
+    finally:
+        Path(path).unlink(missing_ok=True)
+
+    if proc.returncode == 0:
+        return SampleResult("", "", 0, "ok")
+    stderr_tail = proc.stderr.strip().splitlines()[-1] if proc.stderr else ""
+    for exc in ("AttributeError", "TypeError", "ImportError", "NameError"):
+        if re.search(rf"\b{exc}\b", stderr_tail):
+            return SampleResult("", "", 0, exc.lower(), stderr_tail[:200])
+    return SampleResult("", "", 0, "other_error", stderr_tail[:200])
+
+
+def run_task(
+    client: anthropic.Anthropic,
+    task: Task,
+    samples: int,
+    doc: str,
+) -> list[SampleResult]:
+    """Run one task under both conditions for `samples` repetitions each."""
+    results: list[SampleResult] = []
+    prompts = {
+        "baseline": BASELINE_PROMPT.format(module=task.module, task=task.task),
+        "agent_help": TREATMENT_PROMPT.format(
+            module=task.module, doc=doc, target=task.target, task=task.task
+        ),
+    }
+    for condition, prompt in prompts.items():
+        for i in range(samples):
+            result = SampleResult(condition, task.module, i, "no_code")
+            try:
+                code = extract_code(generate(client, prompt))
+                if code:
+                    result = classify(code)
+            except RuntimeError as exc:
+                result = SampleResult(condition, task.module, i, "api_error", str(exc))
+            result.condition = condition
+            result.module = task.module
+            result.sample = i
+            results.append(result)
+            print(
+                f"  {condition:>10} #{i} {task.module}: {result.outcome}"
+                f"{' — ' + result.detail if result.detail else ''}",
+                file=sys.stderr,
+            )
+    return results
+
+
+def summarize(results: list[SampleResult]) -> str:
+    """Render the per-condition outcome table."""
+    by_condition: dict[str, dict[str, int]] = {}
+    for r in results:
+        counts = by_condition.setdefault(r.condition, {})
+        counts[r.outcome] = counts.get(r.outcome, 0) + 1
+
+    outcomes = sorted({r.outcome for r in results})
+    conditions = list(by_condition)
+    header = f"{'outcome':<15}" + "".join(f"{c:>12}" for c in conditions)
+    rows = [
+        f"{outcome:<15}"
+        + "".join(f"{by_condition[c].get(outcome, 0):>12}" for c in conditions)
+        for outcome in outcomes
+    ]
+    total = len(results) // max(len(conditions), 1)
+    footer = f"{'ok rate':<15}" + "".join(
+        f"{by_condition[c].get('ok', 0) / total:>12.0%}" if total else "          0%"
+        for c in conditions
+    )
+    return "\n".join([header, *rows, footer])
+
+
+def load_tasks(path: str | None) -> tuple[Task, ...]:
+    if path is None:
+        return DEFAULT_TASKS
+    raw: Any = json.loads(Path(path).read_text())
+    return tuple(Task(**entry) for entry in raw)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    parser.add_argument("--tasks", help="JSON file of {module, target, task} entries")
+    parser.add_argument(
+        "--samples", type=int, default=3, help="repetitions per condition"
+    )
+    parser.add_argument("--out", help="write full results JSON here")
+    args = parser.parse_args()
+
+    client = anthropic.Anthropic()
+    tasks = load_tasks(args.tasks)
+    all_results: list[SampleResult] = []
+    for task in tasks:
+        print(f"== {task.module} ({task.target}) ==", file=sys.stderr)
+        doc = agent_help(_import_target(task.target))
+        all_results.extend(run_task(client, task, args.samples, doc))
+
+    print(summarize(all_results))
+    if args.out:
+        payload = json.dumps([asdict(r) for r in all_results], indent=2)
+        Path(args.out).write_text(payload)
+        print(f"\nFull results written to {args.out}", file=sys.stderr)
+
+
+def _import_target(dotted: str) -> Any:
+    """Resolve a CLI-style target path to a live object."""
+    from agent_readable.__main__ import _resolve
+
+    return _resolve(dotted)
+
+
+if __name__ == "__main__":
+    main()