answer-engine-benchmark 0.1.0__tar.gz

answer_engine_benchmark-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Synapse Research Ltd
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

answer_engine_benchmark-0.1.0/PKG-INFO

@@ -0,0 +1,180 @@
+Metadata-Version: 2.4
+Name: answer-engine-benchmark
+Version: 0.1.0
+Summary: Ask ChatGPT, Gemini, Perplexity and Claude your buyers' questions, several times each, and count who they cite.
+Author: Synapse Research Ltd
+License-Expression: MIT
+Project-URL: Homepage, https://synapsereality.io/open-source/answer-engine-benchmark/
+Project-URL: Documentation, https://synapsereality.io/open-source/answer-engine-benchmark/
+Project-URL: Repository, https://github.com/synapsereality/answer-engine-benchmark
+Project-URL: Issues, https://github.com/synapsereality/answer-engine-benchmark/issues
+Keywords: aeo,geo,answer-engine-optimization,llm,citations,benchmark,seo
+Classifier: Environment :: Console
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests>=2.28
+Requires-Dist: PyYAML>=6
+Provides-Extra: test
+Requires-Dist: pytest>=8; extra == "test"
+Dynamic: license-file
+
+# answer-engine-benchmark
+
+Asks ChatGPT, Gemini, Perplexity and Claude the questions your buyers ask, with
+web search on, several times each. Then it counts how often each answer cites
+your site, what it cites instead, and what it says about you. Every answer is
+kept in a JSONL file, so you can check any number in the report against the
+text behind it.
+
+Docs: https://synapsereality.io/open-source/answer-engine-benchmark/
+
+```bash
+pip install answer-engine-benchmark
+export OPENAI_API_KEY=... GEMINI_API_KEY=... PERPLEXITY_API_KEY=... ANTHROPIC_API_KEY=...
+aeb run questions/template.yaml --out runs/first \
+  --set brand="Acme Analytics" --set domain=acme.example \
+  --set category="invoice software" --set audience="small accounting firms"
+```
+
+## Why repeat every question
+
+The same question can come back with different sources a minute later. One
+answer is one sample. The default is 3 runs per question per engine, and every
+rate in the report is over those runs. `aeb noise` re-asks a sample of questions
+later and tells you whether a change you see is bigger than the noise.
+
+## The question set
+
+`questions/template.yaml` holds 24 questions in four groups:
+
+| group | the buyer is | your name in the question |
+|---|---|---|
+| discovery | looking for a provider | no |
+| comparison | learning how to choose | no |
+| brand | asking about you | yes |
+| trust | checking you are safe to buy from | yes |
+
+Discovery and comparison tell you whether engines find you when nobody asked
+for you. Brand and trust tell you what they say when somebody does.
+
+Fill the four `vars` in the file, or pass them with `--set`. A run won't start
+while any of them still holds its example value. Add your own questions under
+any group, or new groups. Write them the way a buyer types, and never make a
+competitor the subject of a question.
+
+The file also sets:
+
+- `ours`: domains that count as your citation. `acme.example` covers its
+  subdomains. `github.com/acme` covers only paths under it, so a citation of
+  someone else's GitHub repo is not yours.
+- `brand_terms`: names that count as a mention.
+- `stale_markers`: phrases that describe you wrongly or out of date, such as an
+  old product or a wrong founding year. One is flagged only within 200
+  characters of a brand term, so another company "founded in 2015" doesn't count
+  against you.
+
+## How an answer is scored
+
+**Cited** means one of the answer's URLs is yours. The URLs are the sources the
+engine attached plus any link in the text. **Mentioned** means a brand term
+appears in the text. An answer can mention you without citing you, and that
+difference is worth watching. **Position** is the rank of your first source
+among the distinct sites the answer cited.
+
+Failed calls are recorded as errors and left out of every rate. A rate limit
+never shows up as "not cited".
+
+Gemini returns its sources as Google redirect links. They are resolved to the
+real pages before scoring, with a cache next to the results. Otherwise every
+Gemini answer would look like it cited Google.
+
+## Engines
+
+| engine | key | default model | how it searches |
+|---|---|---|---|
+| `openai` | `OPENAI_API_KEY` | `gpt-5-mini` | Responses API `web_search` tool |
+| `gemini` | `GEMINI_API_KEY` | `gemini-3.5-flash` | Google Search grounding |
+| `perplexity` | `PERPLEXITY_API_KEY` | `perplexity/sonar` | Responses API `web_search` tool |
+| `claude` | `ANTHROPIC_API_KEY` | `claude-sonnet-5` | Messages API web search tool |
+
+Keys are read from the environment only. They are sent in request headers and
+removed from any error message before it is written, so they never reach the
+results file. An engine with no key is skipped.
+
+Change a model with `--model claude=claude-opus-5`. Pick the models your
+buyers actually use in the apps, or say in the report which ones you used.
+
+Perplexity's API doesn't search unless you ask it to. Without the search tool
+it still answers, fluently, with no citations. This adapter always turns search
+on.
+
+## Commands
+
+```bash
+aeb check QUESTIONS [--set ...]           # validate, show which keys are set and how many calls a run needs
+aeb run QUESTIONS --out DIR [--set ...]   # ask everything, write DIR/answers.jsonl and DIR/report.md
+aeb run QUESTIONS --out DIR --dry-run     # first question of each group, once: a cheap smoke test
+aeb report DIR/answers.jsonl [QUESTIONS]  # rebuild the report, re-scoring if you changed the question file
+aeb noise QUESTIONS --baseline DIR/answers.jsonl --out DIR2 --sample 5
+```
+
+`--anonymise` on `run` and `report` replaces every domain that isn't yours with
+"Source A", "Source B" and so on. Use it before you share a report outside your
+company. `--max-calls` (default 500) stops a run that would make more calls
+than you expected.
+
+`answers.jsonl` gets one line per answer, appended as it arrives, so a stopped
+run keeps what it already paid for. Each line holds the question, engine, model,
+run number, the full answer text, every URL, token usage, cost, and the score
+fields.
+
+## What it costs
+
+A full run of the template is 24 questions x 4 engines x 3 runs = 288 calls.
+Our own run of 360 answers cost $1.39 in API fees without Claude (details in
+`examples/`). Adding Claude costs more, because web search on the Claude API is
+billed per search on top of tokens. Each
+row carries its cost. Perplexity reports the real cost. The others are
+estimated from list prices in `engines.py`, so check them against your bills.
+
+## Measure as a stranger
+
+Run the benchmark from an account and machine that has never been told who you
+are. We learned this from a run through a coding assistant's CLI instead of an
+API. The CLI passed the operator's own git identity to the model, and many brand
+answers then told the reader the company was probably their own. The API
+adapters here send only the question and a one-line instruction to cite sources.
+
+## Example
+
+`examples/synapse-launch-week-2026-09.md` is a real run on one company's own
+site, with the published figures only. 105 of 156 answers that named the
+company cited its site. Of the 204 that didn't name it, 1 did. It is an example
+of the output, not part of the question set.
+
+## Install from source
+
+```bash
+git clone https://github.com/synapsereality/answer-engine-benchmark
+cd answer-engine-benchmark
+pip install .
+aeb --help
+```
+
+Python 3.10 or later. Needs `requests` and `PyYAML`.
+
+## Tests
+
+```bash
+pip install -e ".[test]"
+pytest
+```
+
+33 tests. They mock every API, so they spend nothing and need no keys.
+
+## Licence
+
+MIT. Made by [Synapse](https://synapsereality.io/open-source/answer-engine-benchmark/).

answer_engine_benchmark-0.1.0/README.md

@@ -0,0 +1,157 @@
+# answer-engine-benchmark
+
+Asks ChatGPT, Gemini, Perplexity and Claude the questions your buyers ask, with
+web search on, several times each. Then it counts how often each answer cites
+your site, what it cites instead, and what it says about you. Every answer is
+kept in a JSONL file, so you can check any number in the report against the
+text behind it.
+
+Docs: https://synapsereality.io/open-source/answer-engine-benchmark/
+
+```bash
+pip install answer-engine-benchmark
+export OPENAI_API_KEY=... GEMINI_API_KEY=... PERPLEXITY_API_KEY=... ANTHROPIC_API_KEY=...
+aeb run questions/template.yaml --out runs/first \
+  --set brand="Acme Analytics" --set domain=acme.example \
+  --set category="invoice software" --set audience="small accounting firms"
+```
+
+## Why repeat every question
+
+The same question can come back with different sources a minute later. One
+answer is one sample. The default is 3 runs per question per engine, and every
+rate in the report is over those runs. `aeb noise` re-asks a sample of questions
+later and tells you whether a change you see is bigger than the noise.
+
+## The question set
+
+`questions/template.yaml` holds 24 questions in four groups:
+
+| group | the buyer is | your name in the question |
+|---|---|---|
+| discovery | looking for a provider | no |
+| comparison | learning how to choose | no |
+| brand | asking about you | yes |
+| trust | checking you are safe to buy from | yes |
+
+Discovery and comparison tell you whether engines find you when nobody asked
+for you. Brand and trust tell you what they say when somebody does.
+
+Fill the four `vars` in the file, or pass them with `--set`. A run won't start
+while any of them still holds its example value. Add your own questions under
+any group, or new groups. Write them the way a buyer types, and never make a
+competitor the subject of a question.
+
+The file also sets:
+
+- `ours`: domains that count as your citation. `acme.example` covers its
+  subdomains. `github.com/acme` covers only paths under it, so a citation of
+  someone else's GitHub repo is not yours.
+- `brand_terms`: names that count as a mention.
+- `stale_markers`: phrases that describe you wrongly or out of date, such as an
+  old product or a wrong founding year. One is flagged only within 200
+  characters of a brand term, so another company "founded in 2015" doesn't count
+  against you.
+
+## How an answer is scored
+
+**Cited** means one of the answer's URLs is yours. The URLs are the sources the
+engine attached plus any link in the text. **Mentioned** means a brand term
+appears in the text. An answer can mention you without citing you, and that
+difference is worth watching. **Position** is the rank of your first source
+among the distinct sites the answer cited.
+
+Failed calls are recorded as errors and left out of every rate. A rate limit
+never shows up as "not cited".
+
+Gemini returns its sources as Google redirect links. They are resolved to the
+real pages before scoring, with a cache next to the results. Otherwise every
+Gemini answer would look like it cited Google.
+
+## Engines
+
+| engine | key | default model | how it searches |
+|---|---|---|---|
+| `openai` | `OPENAI_API_KEY` | `gpt-5-mini` | Responses API `web_search` tool |
+| `gemini` | `GEMINI_API_KEY` | `gemini-3.5-flash` | Google Search grounding |
+| `perplexity` | `PERPLEXITY_API_KEY` | `perplexity/sonar` | Responses API `web_search` tool |
+| `claude` | `ANTHROPIC_API_KEY` | `claude-sonnet-5` | Messages API web search tool |
+
+Keys are read from the environment only. They are sent in request headers and
+removed from any error message before it is written, so they never reach the
+results file. An engine with no key is skipped.
+
+Change a model with `--model claude=claude-opus-5`. Pick the models your
+buyers actually use in the apps, or say in the report which ones you used.
+
+Perplexity's API doesn't search unless you ask it to. Without the search tool
+it still answers, fluently, with no citations. This adapter always turns search
+on.
+
+## Commands
+
+```bash
+aeb check QUESTIONS [--set ...]           # validate, show which keys are set and how many calls a run needs
+aeb run QUESTIONS --out DIR [--set ...]   # ask everything, write DIR/answers.jsonl and DIR/report.md
+aeb run QUESTIONS --out DIR --dry-run     # first question of each group, once: a cheap smoke test
+aeb report DIR/answers.jsonl [QUESTIONS]  # rebuild the report, re-scoring if you changed the question file
+aeb noise QUESTIONS --baseline DIR/answers.jsonl --out DIR2 --sample 5
+```
+
+`--anonymise` on `run` and `report` replaces every domain that isn't yours with
+"Source A", "Source B" and so on. Use it before you share a report outside your
+company. `--max-calls` (default 500) stops a run that would make more calls
+than you expected.
+
+`answers.jsonl` gets one line per answer, appended as it arrives, so a stopped
+run keeps what it already paid for. Each line holds the question, engine, model,
+run number, the full answer text, every URL, token usage, cost, and the score
+fields.
+
+## What it costs
+
+A full run of the template is 24 questions x 4 engines x 3 runs = 288 calls.
+Our own run of 360 answers cost $1.39 in API fees without Claude (details in
+`examples/`). Adding Claude costs more, because web search on the Claude API is
+billed per search on top of tokens. Each
+row carries its cost. Perplexity reports the real cost. The others are
+estimated from list prices in `engines.py`, so check them against your bills.
+
+## Measure as a stranger
+
+Run the benchmark from an account and machine that has never been told who you
+are. We learned this from a run through a coding assistant's CLI instead of an
+API. The CLI passed the operator's own git identity to the model, and many brand
+answers then told the reader the company was probably their own. The API
+adapters here send only the question and a one-line instruction to cite sources.
+
+## Example
+
+`examples/synapse-launch-week-2026-09.md` is a real run on one company's own
+site, with the published figures only. 105 of 156 answers that named the
+company cited its site. Of the 204 that didn't name it, 1 did. It is an example
+of the output, not part of the question set.
+
+## Install from source
+
+```bash
+git clone https://github.com/synapsereality/answer-engine-benchmark
+cd answer-engine-benchmark
+pip install .
+aeb --help
+```
+
+Python 3.10 or later. Needs `requests` and `PyYAML`.
+
+## Tests
+
+```bash
+pip install -e ".[test]"
+pytest
+```
+
+33 tests. They mock every API, so they spend nothing and need no keys.
+
+## Licence
+
+MIT. Made by [Synapse](https://synapsereality.io/open-source/answer-engine-benchmark/).

answer_engine_benchmark-0.1.0/pyproject.toml

@@ -0,0 +1,43 @@
+[build-system]
+requires = ["setuptools>=69"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "answer-engine-benchmark"
+version = "0.1.0"
+description = "Ask ChatGPT, Gemini, Perplexity and Claude your buyers' questions, several times each, and count who they cite."
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+requires-python = ">=3.10"
+authors = [{ name = "Synapse Research Ltd" }]
+keywords = ["aeo", "geo", "answer-engine-optimization", "llm", "citations", "benchmark", "seo"]
+classifiers = [
+  "Environment :: Console",
+  "Programming Language :: Python :: 3",
+  "Topic :: Internet :: WWW/HTTP :: Indexing/Search",
+]
+dependencies = ["requests>=2.28", "PyYAML>=6"]
+
+[project.optional-dependencies]
+test = ["pytest>=8"]
+
+[project.urls]
+Homepage = "https://synapsereality.io/open-source/answer-engine-benchmark/"
+Documentation = "https://synapsereality.io/open-source/answer-engine-benchmark/"
+Repository = "https://github.com/synapsereality/answer-engine-benchmark"
+Issues = "https://github.com/synapsereality/answer-engine-benchmark/issues"
+
+[project.scripts]
+aeb = "answer_engine_benchmark.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+pythonpath = ["src"]
+testpaths = ["tests"]
+
+[tool.ruff]
+line-length = 120
+target-version = "py310"

answer_engine_benchmark-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

answer_engine_benchmark-0.1.0/src/answer_engine_benchmark/__init__.py

@@ -0,0 +1,3 @@
+"""answer-engine-benchmark: ask AI answer engines your buyers' questions and count who they cite."""
+
+__version__ = "0.1.0"

answer_engine_benchmark-0.1.0/src/answer_engine_benchmark/__main__.py

@@ -0,0 +1,3 @@
+from .cli import main
+
+raise SystemExit(main())

answer_engine_benchmark-0.1.0/src/answer_engine_benchmark/cli.py

@@ -0,0 +1,168 @@
+"""aeb: ask AI answer engines your buyers' questions and count who they cite.
+
+Exit codes: 0 done, 1 a run finished but every call failed, 2 bad input.
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from pathlib import Path
+
+from . import __version__
+from .engines import ENGINES, available
+from .questions import QuestionError, count, load
+from .runner import read_jsonl, rescore, run
+from .summary import compare, pick_questions, render, render_compare
+
+DOCS = "https://synapsereality.io/open-source/answer-engine-benchmark/"
+
+
+def _kv(raw: str) -> tuple[str, str]:
+    k, sep, v = raw.partition("=")
+    if not sep or not k.strip():
+        raise argparse.ArgumentTypeError(f"expected name=value, got {raw!r}")
+    return k.strip(), v.strip()
+
+
+def _common(p: argparse.ArgumentParser) -> None:
+    p.add_argument("questions", help="question file (YAML)")
+    p.add_argument("--set", action="append", type=_kv, default=[], metavar="NAME=VALUE",
+                   help="fill a {placeholder}, e.g. --set brand='Acme Ltd' (repeatable)")
+
+
+def _engine_flags(p: argparse.ArgumentParser) -> None:
+    p.add_argument("--engine", action="append", choices=sorted(ENGINES), help="only these engines (repeatable)")
+    p.add_argument("--model", action="append", type=_kv, default=[], metavar="ENGINE=MODEL",
+                   help="override a model, e.g. --model claude=claude-opus-5")
+    p.add_argument("--runs", type=int, default=3, help="times to ask each question on each engine (default 3)")
+    p.add_argument("--max-calls", type=int, default=500,
+                   help="refuse to start a run that needs more API calls than this (default 500)")
+    p.add_argument("--sleep", type=float, default=1.0, help="seconds between calls (default 1)")
+
+
+def build_parser() -> argparse.ArgumentParser:
+    ap = argparse.ArgumentParser(prog="aeb", description=__doc__, epilog=f"Docs: {DOCS}",
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--version", action="version", version=f"aeb {__version__}")
+    sub = ap.add_subparsers(dest="cmd", required=True)
+
+    p = sub.add_parser("check", help="validate a question file and show what a run would cost in calls")
+    _common(p)
+    _engine_flags(p)
+    p.add_argument("--allow-unfilled", action="store_true", help="accept the template's example values")
+
+    p = sub.add_parser("run", help="ask every question on every engine and write answers.jsonl + report.md")
+    _common(p)
+    _engine_flags(p)
+    p.add_argument("--out", required=True, type=Path, help="output folder")
+    p.add_argument("--group", action="append", help="only these question groups (repeatable)")
+    p.add_argument("--dry-run", action="store_true", help="first question of each group, 1 run: a cheap smoke test")
+    p.add_argument("--anonymise", action="store_true", help="label other domains Source A, B, ... in report.md")
+
+    p = sub.add_parser("report", help="rebuild report.md from answers.jsonl (spends nothing)")
+    p.add_argument("answers", type=Path, help="answers.jsonl")
+    p.add_argument("questions", nargs="?", help="question file: re-score with its ours/brand_terms first")
+    p.add_argument("--set", action="append", type=_kv, default=[], metavar="NAME=VALUE")
+    p.add_argument("--anonymise", action="store_true")
+    p.add_argument("--out", type=Path, help="write here instead of stdout")
+
+    p = sub.add_parser("noise", help="re-ask a sample of questions and compare with the baseline")
+    _common(p)
+    _engine_flags(p)
+    p.add_argument("--baseline", required=True, type=Path, help="answers.jsonl from the first run")
+    p.add_argument("--out", required=True, type=Path, help="output folder for the repeat run")
+    p.add_argument("--sample", type=int, default=5, help="questions to re-ask, spread across groups (default 5)")
+    p.add_argument("--tolerance", type=int, default=1, help="allowed difference in citations (default 1)")
+    return ap
+
+
+def _plan(q: dict, engines: dict, runs: int, max_calls: int) -> int | None:
+    calls = count(q) * len(engines) * runs
+    names = ", ".join(f"{n} ({e.model})" for n, e in engines.items()) or "none"
+    print(f"{count(q)} questions x {len(engines)} engines x {runs} runs = {calls} calls. Engines: {names}")
+    if calls > max_calls:
+        print(f"aeb: {calls} calls is over --max-calls {max_calls}. Raise it if you mean it.", file=sys.stderr)
+        return None
+    return calls
+
+
+def _progress(rec: dict) -> None:
+    flag = "ERR " if rec["error"] else "CITE" if rec["cited"] else "ment" if rec["mentioned"] else "  - "
+    cost = rec["cost_usd"] or 0
+    print(f"{flag} {rec['engine']:<10} {rec['group']:<11} pos={rec['position'] or '-':<3} ${cost:.4f} "
+          f"{rec['question'][:70]} {rec['error'][:80]}", flush=True)
+
+
+def main(argv: list[str] | None = None) -> int:
+    args = build_parser().parse_args(argv)
+    try:
+        if args.cmd == "report":
+            recs = read_jsonl(args.answers)
+            ours = []
+            if args.questions:
+                q = load(args.questions, dict(args.set), allow_unfilled=True)
+                recs, ours = rescore(recs, q), q["ours"]
+            text = render(recs, ours, anonymise=args.anonymise)
+            if args.out:
+                args.out.write_text(text, encoding="utf-8")
+            else:
+                print(text, end="")
+            return 0
+
+        q = load(args.questions, dict(args.set), allow_unfilled=getattr(args, "allow_unfilled", False))
+        engines = available(only=args.engine, models=dict(args.model))
+        if args.cmd == "check":
+            print(f"ok: {count(q)} questions in {len(q['groups'])} groups; ours = {q['ours']}")
+            missing = [f"{n} ({e.env_key})" for n, e in ENGINES.items()
+                       if n not in engines and (not args.engine or n in args.engine)]
+            if missing:
+                print("no key set for: " + ", ".join(missing))
+            _plan(q, engines, args.runs, args.max_calls)
+            return 0
+
+        if not engines:
+            keys = ", ".join(e.env_key for e in ENGINES.values())
+            print(f"aeb: no engine has a key. Set one or more of: {keys}", file=sys.stderr)
+            return 2
+
+        if args.cmd == "run":
+            if args.group:
+                q["groups"] = {g: v for g, v in q["groups"].items() if g in args.group}
+            if args.dry_run:
+                q["groups"] = {g: v[:1] for g, v in q["groups"].items()}
+                args.runs = 1
+            if _plan(q, engines, args.runs, args.max_calls) is None:
+                return 2
+            answers = args.out / "answers.jsonl"
+            recs = run(q, engines, runs=args.runs, out=answers, on_result=_progress, sleep=args.sleep)
+            everything = read_jsonl(answers)
+            (args.out / "report.md").write_text(render(everything, q["ours"], anonymise=args.anonymise),
+                                                encoding="utf-8")
+            cost = sum(r.get("cost_usd") or 0 for r in recs)
+            print(f"wrote {answers} (+{len(recs)} answers) and report.md. This run cost about ${cost:.3f}")
+            return 1 if recs and all(r["error"] for r in recs) else 0
+
+        if args.cmd == "noise":
+            baseline = read_jsonl(args.baseline)
+            picked = pick_questions(q, args.sample)
+            groups: dict[str, list[str]] = {}
+            for g, question in picked:
+                groups.setdefault(g, []).append(question)
+            q["groups"] = groups
+            if _plan(q, engines, args.runs, args.max_calls) is None:
+                return 2
+            answers = args.out / "repeat.jsonl"
+            run(q, engines, runs=args.runs, out=answers, on_result=_progress, sleep=args.sleep, label="repeat")
+            rows = compare(baseline, read_jsonl(answers), tolerance=args.tolerance)
+            text = render_compare(rows, args.tolerance)
+            (args.out / "noise.md").write_text(text, encoding="utf-8")
+            print(text, end="")
+            return 0
+    except QuestionError as e:
+        print(f"aeb: {e}", file=sys.stderr)
+        return 2
+    except (OSError, ValueError) as e:
+        print(f"aeb: {e}", file=sys.stderr)
+        return 2
+    return 2