pasr-bench 0.2.0__py3-none-any.whl

pasr_bench-0.2.0.dist-info/METADATA

@@ -0,0 +1,116 @@
+Metadata-Version: 2.5
+Name: pasr-bench
+Version: 0.2.0
+Summary: PASR-Bench: a pre-registered protocol for measuring a code-context retriever against grep, a repo-map, semantic search, and a whole-repo dump.
+Project-URL: Homepage, https://github.com/Apheironn/pasr
+Project-URL: Source, https://github.com/Apheironn/pasr/tree/main/eval
+Author: Apheironn
+License-Expression: Apache-2.0
+Keywords: benchmark,coding-agent,context,evaluation,non-inferiority,retrieval
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Requires-Dist: pasr-mcp>=0.2
+Provides-Extra: dev
+Requires-Dist: pytest<9,>=8; extra == 'dev'
+Requires-Dist: ruff>=0.6; extra == 'dev'
+Provides-Extra: llm
+Requires-Dist: anthropic>=0.40; extra == 'llm'
+Provides-Extra: plots
+Requires-Dist: matplotlib>=3.7; extra == 'plots'
+Description-Content-Type: text/markdown
+
+# PASR-Bench
+
+A small, **pre-registered** protocol for measuring a code-context retriever, and the
+harness that runs it. Packaged separately from `pasr-mcp` (`pip install
+./eval`, distribution name `pasr-bench`). Results for PASR itself:
+[`RESULTS.md`](RESULTS.md) and [`../docs/competitors-benchmark.md`](../docs/competitors-benchmark.md).
+
+## What it measures
+
+**Real-agent run** — four arms per task, each producing the same `ArmResult`:
+
+1. **native_search** — deterministic grep + read-files (stand-in for an agent's own tools)
+2. **broad** — the whole repo, source-first, truncated to a cap
+3. **pasr** — `select_context` at a fixed budget
+4. **pasr_fallback** — `pasr`, then one budget widening if confidence is low
+
+A model answers each task from **only** that arm's context (`UNKNOWN` if absent); a
+second model judges the answer against the task's expected identifiers;
+`critical_source_hit` is still required. Metrics: task success, model input tokens
+(incl. a flat per-tool-call overhead), round trips, critical-source miss rate, fallback
+rate. Then a **paired non-inferiority** test of `pasr` / `pasr_fallback` vs the
+`baseline_arm` at the plan's `margin_task_success` (point estimate + 10k-resample
+bootstrap CI).
+
+**Bake-off** — an offline, no-API retrieval comparison at a shared budget: `grep`,
+`repomap` (aider-style signatures), `embed_lex` (a no-setup semantic floor), `pasr`,
+`pasr_hash`, `pasr_map`. Scored on critical-file hit ∧ keyword coverage, split by task
+kind.
+
+**No GPU.** Arms run on CPU; the only model use is one answer + one judge call per
+(task, arm) — ~400 Anthropic calls for the 50-task plan. Full Sonnet ≈ $12–15; a cheap
+`--model` with a strong `--judge-model` ≈ $4–6.
+
+## Install & run
+
+```bash
+pip install ./eval               # the pasr-bench distribution (deps: pasr-mcp)
+pip install "./eval[llm,plots]"  # + anthropic (real agent) + matplotlib (report.png)
+
+pasr-bench plans                             # the packaged plan(s)
+pasr-bench run --agent keyword               # offline smoke: full matrix + report + validation
+pasr-bench bakeoff --budget 6000             # offline retrieval bake-off
+
+export ANTHROPIC_API_KEY=sk-ant-...
+pasr-bench run --agent claude --max-tasks 4  # cheap trial (~$0.4)
+pasr-bench run --agent claude \
+  --model claude-haiku-4-5 --judge-model claude-sonnet-5 \
+  --checkout-dir .eval-checkouts             # budget-safe full run (~$4–6)
+```
+
+Each finished `(task, arm)` row is appended to `matrix.jsonl` and flushed, so a crash
+keeps every completed row; `--resume <run_dir>` reloads the partial matrix and finishes
+into the same delivery. `--checkout-dir DIR` reuses clones. `PASR_EVAL_BROAD_CAP=30000`
+shrinks the `broad` arm. A run writes `<out>/<plan>_<utc>/` with `matrix.jsonl`,
+`report.{json,md,png}`, `validation.json`, `resolved_commits.json`.
+
+`python eval/run_eval.py …` and `python eval/bakeoff.py …` still work as thin shims for
+the two sub-commands.
+
+## Bring your own retriever
+
+The protocol is retriever-agnostic. To measure a different context tool against the
+same 50 tasks and the same baselines:
+
+1. Add an arm in [`pasr_eval/arms.py`](pasr_eval/arms.py): extend `ARMS` and add a
+   branch in `_build(...)` that returns `(context, sources, tool_calls, round_trips,
+   fallback)` for your retriever. Everything downstream — grading, metrics,
+   non-inferiority, the validator — is arm-agnostic.
+2. For a bake-off arm, add a function in [`pasr_eval/bakeoff.py`](pasr_eval/bakeoff.py)
+   and list it in that file's `ARMS`.
+3. Keep the plan (`pasr_eval/plans/pilot.json`) fixed so numbers stay comparable, or
+   register a new plan and cite it.
+
+The validator (`validate_matrix`) rejects synthetic rows, query→answer leaks, and an
+unmatched task×arm matrix, so a submitted result is checkable.
+
+## Files
+
+| Path | What |
+|---|---|
+| `pasr_eval/spec.py` | `RepoSpec` / `TaskSpec` / `EvalPlan`, `load_plan`; the leak + kind guards |
+| `pasr_eval/arms.py` | the four real-agent arms → `ArmResult` |
+| `pasr_eval/bakeoff.py` | the six offline bake-off arms |
+| `pasr_eval/agents.py` | `AgentRunner` protocol, `KeywordAgent` (offline proxy) |
+| `pasr_eval/llm_agent.py` | `LlmAgent` — answer + judge via the Anthropic API (`[llm]` extra) |
+| `pasr_eval/metrics.py` | grade, aggregate, paired bootstrap CI, non-inferiority, `full_report` |
+| `pasr_eval/runner.py` | `resolve_repos`, `run_plan` (`skip=` / `on_row=`), `write_matrix` |
+| `pasr_eval/validate.py` | `validate_matrix` — no synthetic rows, no leaks, matched matrix |
+| `pasr_eval/run.py` | end-to-end orchestrator: clone → streamed matrix → report → validate |
+| `pasr_eval/plans/pilot.json` | the registered plan — 10 pinned repos, 50 tasks |
+| `RESULTS.md` | pre-registration + n=15 pilot + keyword-50 + the n=50 real-agent headline |

pasr_bench-0.2.0.dist-info/RECORD

@@ -0,0 +1,17 @@
+pasr_eval/__init__.py,sha256=AdMWfjzTeqOuI9_8t2NMNTTpBMk9UgBM-7R8sP0u-Pk,1156
+pasr_eval/__main__.py,sha256=nkasG0AYtek3_6AGmws07vWKKYX-Uun2-5x5mbAoa-g,1578
+pasr_eval/agents.py,sha256=jyqz-BN4SWbIwseztkZw11v0aIXurMVoNaLd-xDuTec,931
+pasr_eval/arms.py,sha256=Q6q87VzvFbxQINc_rTk-to0iNW0Ms4B94Yzae_UwvQI,6857
+pasr_eval/bakeoff.py,sha256=KqzBPCfz6YUJ-Uw-B3evscMm2TfR-yxsbmmWoks6CwM,13197
+pasr_eval/llm_agent.py,sha256=ksxd2l70YK6z5w0L4Igq9lpoq2jt530otsOw-JoAHAQ,2625
+pasr_eval/metrics.py,sha256=e8uF_qGBl99wrNp1Y8nmIYj9r11pvbjEtIUh6snPJ_g,4119
+pasr_eval/run.py,sha256=LJeDhfSfNr0LXsG-19S7kV4kAyq3p3D-uEp2Ppklsp4,7967
+pasr_eval/runner.py,sha256=j_eMA6_ohIpmvr4cVRMLWzFnHJ2iflZhb3rL1ZfgTSo,3676
+pasr_eval/spec.py,sha256=H9bfYlaTwBFN1NRqWy-6r2_sxjTQCoM3syHA_pm6DUE,4362
+pasr_eval/validate.py,sha256=HaikrR4qAzE-94-GG5k01aFPW2x9wRUMi3nSdkESUU0,2963
+pasr_eval/plans/pilot.json,sha256=zOBS7Hac3xj_g86cgwCciE0K8ogCpo9xpFojcOe9IF8,13463
+pasr_eval/RESULTS.md,sha256=UURZqb52gaf72QTVmZe_UlEHTA1YzEqduUJNNAtkXfQ,9299
+pasr_bench-0.2.0.dist-info/METADATA,sha256=djPvb6EK07FtjgfPl-_rKNI5gPXkBC8z5jwLtypB4lw,5889
+pasr_bench-0.2.0.dist-info/WHEEL,sha256=zOwg4jB6zX2kU910N-cMawjivD6tO8NEWvE12je1bVk,87
+pasr_bench-0.2.0.dist-info/entry_points.txt,sha256=llKv_M2edVivOvKiYPqcl_OTRaaXHVG18_9ObB1cBuU,55
+pasr_bench-0.2.0.dist-info/RECORD,,

pasr_bench-0.2.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.32.0
+Root-Is-Purelib: true
+Tag: py3-none-any

pasr_bench-0.2.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+pasr-bench = pasr_eval.__main__:main

pasr_eval/RESULTS.md

@@ -0,0 +1,151 @@
+# PASR real-agent evaluation — results
+
+> **50-task real-agent run done** (`eval/deliveries/realagent50_20260902T221528Z/`,
+> 10 pinned repos, Claude answer + judge). PASR gives the model **5.8k targeted
+> tokens in one tool call** and it answers **48%** of the tasks; the **59k-token
+> source-first repo dump answers 38%** — PASR is **+0.10** on paired task success
+> (`pasr_fallback` +0.12), and the **point estimate clears the -0.05
+> non-inferiority margin** (95% CI still crosses it: `pasr` [-0.08, +0.30]).
+> PASR puts the critical file in context **46/50** times vs the dump's **33/50**.
+>
+> Reading: on localized code questions PASR matches — slightly beats — a 10×-larger
+> whole-repo dump, at **~90% fewer input tokens and one round trip**. `native_search`
+> (grep + read 6 files, 23k tokens, 6 round trips) is the raw-success leader at 0.52
+> but misses the critical file 30% of the time. This is a **bounded efficiency
+> result**, not a superiority claim — one more batch of 50 would settle the interval.
+
+## Pre-registration
+
+- **Plan:** `pasr_eval/plans/pilot.json` — 10 pinned public Python repos, 50 source-grounded
+  `locate` / `trace` / `explain` tasks (5/repo). `pasr-bench run` records each repo's
+  resolved commit SHA and the answer/judge model IDs into `matrix.jsonl`'s `_meta` line.
+- **Arms:** `native_search`, `broad`, `pasr`, `pasr_fallback`.
+- **Baseline:** `broad`.
+- **Primary claim (non-inferiority):** on paired tasks, `pasr_fallback` task success
+  is non-inferior to `broad` within a **-0.05** margin (both the point estimate and
+  the lower bound of a 10k-resample paired bootstrap 95% CI ≥ margin).
+- **Secondary:** `pasr` / `pasr_fallback` reduce mean `tokens_in` and `round_trips`
+  vs `broad`; `critical_source_miss_rate` reported per arm; `fallback_rate` reported.
+- **No synthetic rows, no reference leak:** enforced by `validate_matrix` (every row
+  traces to a registered task + repo; the query never contains the full answer;
+  exactly one row per task×arm).
+
+## The two graders
+
+- **Keyword proxy** (`--agent keyword`, offline, no API): task success = every expected
+  identifier appears verbatim in the arm's context **and** the critical file is present.
+  A harsh literal proxy for retrieval quality — machinery + token-story check only.
+- **Real agent** (`--agent claude`): a model answers from **only** the arm's context
+  (`UNKNOWN` if not present), then a second model judges the answer against the task's
+  expected identifiers; `critical_source_hit` is still required for a pass.
+
+The n=15 pilot and the 50-task keyword re-run below are the supporting runs; the
+[50-task real-agent run](#50-task-real-agent-run--claude-answer--judge-10-repos---headline)
+is the headline.
+
+## Pilot run — `claude-sonnet-5`, 15 tasks, 5 repos
+
+| arm | task success | context tokens | round trips | crit. miss | fallback rate |
+|---|---:|---:|---:|---:|---:|
+| broad (58k source-first repo dump) | 0.467 | 58 146 | 1 | 0.067 | – |
+| native_search (grep + 6 files, 4k each) | 0.400 | 22 771 | 6 | 0.267 | – |
+| **pasr** | **0.533** | **5 764** | **1** | 0.133 | – |
+| pasr_fallback | 0.600 | 5 764 | 1 | 0.133 | 0.000 |
+
+- **Token savings vs `broad`: pasr / pasr_fallback +90.0%**; native_search +60.4%.
+- **Non-inferiority (`pasr_fallback` vs `broad`, task success, margin -0.05):**
+  delta **+0.133**, 95% CI **[-0.20, +0.47]** — **point estimate PASSES**, the CI
+  lower bound (-0.20) does **not** clear the margin. `pasr` vs `broad`: delta +0.067,
+  CI [-0.33, +0.47], same picture.
+- `pasr` critical-source miss: **2/15** (`req-02`, `star-03`; `star-03` was also
+  missed by `broad` → likely a task-spec problem, not retrieval).
+- The `pasr_fallback` threshold (`confidence < 0.5`) **never triggered** — all 15
+  PASR selections were confident. `pasr_fallback` ran on the same contexts as `pasr`;
+  its +0.067 over `pasr` is answer/judge sampling noise, not a fallback effect.
+
+### Reading it
+
+- `broad` puts the critical file in context 14/15 times (58k tokens) yet the model
+  answers only 7/15 — a large-haystack utilisation effect. `pasr` gives the model
+  5.8k targeted tokens and it answers 8/15. **Same answer quality band, one order of
+  magnitude fewer tokens, one tool call instead of the model chewing 58k.**
+- `native_search` is worst: brittle grep (27% critical miss) plus six partial files.
+- **What this does not show:** inferential non-inferiority (n=15 CI is ±0.3). The
+  bounded positive claim is a cross-repo efficiency direction, mirroring `researchv2`'s
+  LongBench Pro finding.
+
+Delivery: `eval/deliveries/pilot_20260902T212108Z/`.
+
+## 50-task keyword-proxy re-run (no API — `pasr-bench run --agent keyword`)
+
+`eval/deliveries/keyword50_20260902T213744Z/`. 10 repos, 50 source-grounded tasks;
+`_FALLBACK_CONFIDENCE` raised to 0.65.
+
+| arm | task success | context tokens | round trips | crit. miss | fallback rate |
+|---|---:|---:|---:|---:|---:|
+| broad (58k source-first) | 0.660 | 59 075 | 1 | **0.340** | – |
+| native_search (6 files, 4k) | 0.640 | 22 901 | 6 | 0.300 | – |
+| **pasr** | **0.700** | **5 777** | 1 | **0.080** | – |
+| pasr_fallback | 0.700 | 5 854 | 1.02 | 0.060 | 0.020 |
+
+- **PASR beats both baselines on the literal grader** (0.70 vs 0.66 / 0.64) at **+90%
+  tokens**, and its critical-source miss (0.08 = 4/50) is **4× lower than broad's
+  0.34** — a 60k source-first dump still fails to include the right file for 17/50
+  real-repo tasks; PASR's targeted retrieval gets it in 92% of the time.
+- Non-inferiority (`pasr` vs `broad`, keyword grader): delta **+0.04**, 95% CI
+  **[-0.14, +0.22]** — point PASSES, CI is now ±0.18 (was ±0.30 at n=15) and just
+  misses clearing the -0.05 margin.
+- Fallback engaged on 1/50 (crit-miss 0.08 → 0.06).
+
+## 50-task real-agent run — Claude answer + judge, 10 repos  ★ headline
+
+`eval/deliveries/realagent50_20260902T221528Z/`. Models recorded in `matrix.jsonl`'s
+`_meta`. `validate_matrix` clean (no synthetic rows, no leak, matched 4×50 matrix).
+
+| arm | task success | context tokens | tokens_in | round trips | crit. miss | fallback |
+|---|---:|---:|---:|---:|---:|---:|
+| broad (59k source-first repo dump) | 0.380 | 59 075 | 59 115 | 1 | **0.340** | – |
+| native_search (grep + 6 files, 4k each) | **0.520** | 22 901 | 23 181 | 6 | 0.300 | – |
+| **pasr** | 0.480 | **5 777** | **5 817** | **1** | **0.080** | – |
+| pasr_fallback | 0.500 | 5 854 | 5 895 | 1.02 | 0.060 | 0.020 |
+
+- **Token savings vs `broad`: pasr +90.2%, pasr_fallback +90.0%**; native_search +60.8%.
+- **Non-inferiority (task success, margin -0.05):** `pasr` delta **+0.100**, 95% CI
+  **[-0.08, +0.30]**; `pasr_fallback` delta **+0.120**, CI **[-0.06, +0.30]**. Both
+  **point estimates PASS**; both CI lower bounds miss the margin by ≈0.03 (half-width
+  ±0.18–0.19 at n=50, down from ±0.30 at n=15).
+- **Critical-source hit: `pasr` 46/50 (92%), `pasr_fallback` 47/50** vs **`broad`
+  33/50 (66%)**. A 59k source-first dump *still* omits the answer's file for 17/50
+  real-repo tasks — concentrated in the large repos (`typer` 5/5 missed, `jinja` 4/5,
+  `packaging` 4/5, `anyio` 3/5); those are exactly the repos where `broad` scores
+  0–1/5. Where the file *does* fit (`requests`, `pluggy`, `httpx`) `broad` reaches
+  4/5. **`broad`'s 0.38 is a truncation + large-haystack failure, not a grading one.**
+- Head-to-head: `pasr` wins **15** tasks `broad` loses, loses **10** `broad` wins
+  (net +5 of 50 = the +0.10). `pasr_fallback` widened once (`pkg`-family), turning one
+  loss into a win.
+- **PASR's weak spot — `typer` (0/5, both PASR arms).** The slice reaches the right
+  file 3/5 but the answering lines aren't in the selected window: `typer` leans on
+  re-exports and decorator plumbing that the current chunker + 6k budget don't
+  resolve. Drop `typer` and `pasr` is 24/45 = **0.53**. Logged for a chunker follow-up.
+- `native_search` is the raw-success leader (0.52) but pays **4× the tokens, 6 round
+  trips**, and a **30% critical-source miss** — brittle when the query terms don't
+  literally appear near the answer.
+
+### Reading it
+
+The claim PASR makes is a **profile trade**, and the run supports it: at parity-ish
+answer quality with a whole-repo dump (−0 to +0.12 depending on arm/margin), PASR
+costs **one order of magnitude fewer input tokens, one tool call instead of the model
+chewing 59k, and a 92% critical-file hit rate with per-line provenance**. It is *not*
+an inferential non-inferiority pass (CI lower bound −0.08) and *not* a raw-accuracy
+win over an agent's own grep. It mirrors `researchv2`'s LongBench Pro finding: a
+bounded cross-repo efficiency direction.
+
+### Next
+
+1. **Second batch of 50 tasks** to close the CI (projected half-width ≈±0.13 at
+   n=100 would clear the −0.05 margin if the point estimate holds).
+2. **Chunker follow-up for re-export/decorator-heavy repos** (`typer`): the pass that
+   turns a critical-file hit into an answerable window.
+3. Consider a relevance-ranked `broad` truncation so it isn't a strawman past ~40k —
+   though the point of this arm is precisely "what a naive big-context dump gets you".

pasr_eval/__init__.py

@@ -0,0 +1,34 @@
+"""Real-agent evaluation harness for the PASR MCP (roadmap M11 / research Track 2).
+
+Not shipped with ``pasr-mcp``. Compares four arms per task — the agent's native
+search, full/broad context, PASR selected context, PASR + one controlled fallback —
+and reports paired metrics with a pre-registered non-inferiority margin.
+
+The dry-run path (``KeywordAgent`` + local-mode plan) exercises the whole matrix
+offline; the real run swaps in ``LlmAgent`` (answer + judge over the Anthropic API).
+"""
+
+from pasr_eval.arms import ARMS, ArmResult, run_arm
+from pasr_eval.metrics import aggregate, bootstrap_ci, grade, non_inferiority, paired_delta
+from pasr_eval.runner import run_plan, write_matrix
+from pasr_eval.spec import EvalPlan, RepoSpec, TaskSpec, load_plan, plan_from_dict
+from pasr_eval.validate import validate_matrix
+
+__all__ = [
+    "ARMS",
+    "ArmResult",
+    "run_arm",
+    "run_plan",
+    "write_matrix",
+    "EvalPlan",
+    "RepoSpec",
+    "TaskSpec",
+    "load_plan",
+    "plan_from_dict",
+    "aggregate",
+    "paired_delta",
+    "bootstrap_ci",
+    "non_inferiority",
+    "grade",
+    "validate_matrix",
+]

pasr_eval/__main__.py

@@ -0,0 +1,47 @@
+"""``pasr-bench`` console entry: a thin dispatcher over the two orchestrators.
+
+    pasr-bench run [args...]       # the 4-arm real-agent / keyword evaluation
+    pasr-bench bakeoff [args...]   # the offline retrieval bake-off
+    pasr-bench plans               # list the packaged plans
+
+Everything after the sub-command is forwarded verbatim, so
+``pasr-bench run --agent claude --resume ...`` works exactly like the old
+``python eval/run_eval.py``.
+"""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+_USAGE = "usage: pasr-bench {run|bakeoff|plans} [args...]"
+
+
+def main(argv: list[str] | None = None) -> int:
+    args = list(sys.argv[1:] if argv is None else argv)
+    if not args or args[0] in ("-h", "--help"):
+        print(_USAGE)
+        print("\n  run      end-to-end evaluation -> a timestamped delivery")
+        print("  bakeoff  offline retrieval bake-off (grep / repo-map / semantic / PASR)")
+        print("  plans    list the plan files bundled with this package")
+        return 0 if args else 2
+
+    cmd, rest = args[0], args[1:]
+    if cmd == "run":
+        from pasr_eval.run import main as run_main
+
+        return run_main(rest)
+    if cmd == "bakeoff":
+        from pasr_eval.bakeoff import main as bakeoff_main
+
+        return bakeoff_main(rest)
+    if cmd == "plans":
+        for path in sorted((Path(__file__).parent / "plans").glob("*.json")):
+            print(path)
+        return 0
+    print(f"unknown sub-command {cmd!r}\n{_USAGE}", file=sys.stderr)
+    return 2
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

pasr_eval/agents.py

@@ -0,0 +1,28 @@
+"""Agent adapters. The offline dry-run uses a deterministic keyword agent; the real
+run uses :class:`pasr_eval.llm_agent.LlmAgent` (answer + judge over the Anthropic API)."""
+
+from __future__ import annotations
+
+from typing import Protocol, runtime_checkable
+
+from pasr_eval.spec import TaskSpec
+
+
+@runtime_checkable
+class AgentRunner(Protocol):
+    name: str
+
+    def answer(self, task: TaskSpec, context: str) -> str:
+        """Answer ``task`` given only ``context`` (the arm's provided slice)."""
+
+
+class KeywordAgent:
+    """Deterministic stand-in: 'answers' with the answer keywords it can ground in
+    the context. No LLM — a retrieval-quality proxy for the offline dry-run."""
+
+    name = "keyword"
+
+    def answer(self, task: TaskSpec, context: str) -> str:
+        low = context.casefold()
+        grounded = [keyword for keyword in task.answer_keywords if keyword.casefold() in low]
+        return " ".join(grounded)

pasr_eval/arms.py

@@ -0,0 +1,195 @@
+"""The four evaluation arms.
+
+Every arm produces the same :class:`ArmResult`: what context the agent got, how many
+tokens / tool calls that cost, whether the critical source made it in, and whether the
+agent could then answer. Deterministic.
+"""
+
+from __future__ import annotations
+
+import os
+from dataclasses import asdict, dataclass, fields
+from pathlib import Path
+
+from pasr.evidence import extract_keywords
+from pasr.file_discovery import discover_workspace_files
+from pasr.schema import validate_select_context_request
+from pasr.select import run_select_context
+from pasr.tokenize import Tokenizer, get_tokenizer
+from pasr_eval.agents import AgentRunner
+from pasr_eval.metrics import grade
+from pasr_eval.spec import TaskSpec
+
+ARMS = ("native_search", "broad", "pasr", "pasr_fallback")
+
+_TOOL_OVERHEAD_TOKENS = 40  # flat per-tool-interaction cost estimate
+_NATIVE_MAX_FILES = 6
+_NATIVE_FILE_CAP_TOKENS = 4000  # an agent reads a big file in ranges, not whole
+# a realistic large-context baseline; PASR_EVAL_BROAD_CAP lets a tight API budget shrink it
+_BROAD_CAP_TOKENS = int(os.environ.get("PASR_EVAL_BROAD_CAP", "60000"))
+_PASR_BUDGET = 6000
+_FALLBACK_EXTRA = 4000
+_FALLBACK_CONFIDENCE = 0.65  # widen once when PASR is not confident in the slice
+
+_TEST_HINTS = ("/test", "test_", "_test.", "/tests/", "conftest")
+_DOC_EXT = (".md", ".rst", ".txt")
+
+
+def _priority(relative_path: str) -> tuple[int, str]:
+    """Source first, then non-test, then non-doc — so budget truncation keeps code."""
+    low = relative_path.casefold()
+    is_test = any(hint in low for hint in _TEST_HINTS)
+    is_doc = low.endswith(_DOC_EXT)
+    return (int(is_test) * 2 + int(is_doc), relative_path)
+
+
+@dataclass(frozen=True)
+class ArmResult:
+    arm: str
+    task_id: str
+    repo: str
+    context_tokens: int
+    tokens_in: int
+    tool_calls: int
+    round_trips: int
+    sources_included: tuple[str, ...]
+    critical_source_hit: bool
+    fallback_triggered: bool
+    answer: str
+    task_success: bool
+
+    def to_dict(self) -> dict:
+        return {**asdict(self), "sources_included": list(self.sources_included)}
+
+    @classmethod
+    def from_dict(cls, data: dict) -> ArmResult:
+        names = {f.name for f in fields(cls)}
+        payload = {k: v for k, v in data.items() if k in names}
+        payload["sources_included"] = tuple(payload.get("sources_included", ()))
+        return cls(**payload)
+
+
+def run_arm(
+    arm: str,
+    task: TaskSpec,
+    repo_root: Path,
+    agent: AgentRunner,
+    tokenizer: Tokenizer | None = None,
+) -> ArmResult:
+    if arm not in ARMS:
+        raise ValueError(f"unknown arm: {arm!r}")
+    tok = tokenizer or get_tokenizer()
+    context, sources, tool_calls, round_trips, fallback = _build(arm, task, repo_root, tok)
+
+    context_tokens = tok.count(context)
+    hit = not task.critical_source or any(task.critical_source in source for source in sources)
+    answer = agent.answer(task, context)
+
+    judge = getattr(agent, "judge", None)
+    success = bool(judge(task, answer, hit)) if callable(judge) else grade(task, answer, _Probe(hit))
+
+    return ArmResult(
+        arm=arm,
+        task_id=task.id,
+        repo=task.repo,
+        context_tokens=context_tokens,
+        tokens_in=context_tokens + tool_calls * _TOOL_OVERHEAD_TOKENS,
+        tool_calls=tool_calls,
+        round_trips=round_trips,
+        sources_included=tuple(sorted(sources)),
+        critical_source_hit=hit,
+        fallback_triggered=fallback,
+        answer=answer,
+        task_success=success,
+    )
+
+
+class _Probe:
+    """Minimal object grade() needs: just ``critical_source_hit``."""
+
+    def __init__(self, hit: bool) -> None:
+        self.critical_source_hit = hit
+
+
+def _build(arm, task, repo_root, tok):
+    if arm == "native_search":
+        return _native_search(task, repo_root, tok)
+    if arm == "broad":
+        return _broad(repo_root, tok)
+    if arm == "pasr":
+        context, sources = _pasr(task, repo_root, budget=_PASR_BUDGET)
+        return context, sources, 1, 1, False
+    # pasr_fallback
+    context, sources = _pasr(task, repo_root, budget=_PASR_BUDGET)
+    result = _pasr_full(task, repo_root, _PASR_BUDGET)
+    if float(result.get("confidence") or 0.0) < _FALLBACK_CONFIDENCE:
+        context, sources = _pasr(task, repo_root, budget=_PASR_BUDGET + _FALLBACK_EXTRA)
+        return context, sources, 2, 2, True
+    return context, sources, 1, 1, False
+
+
+def _discovered(repo_root: Path) -> list:
+    return discover_workspace_files(repo_root, include_patterns=["."])
+
+
+def _native_search(task, repo_root, tok):
+    terms = [t.casefold() for t in extract_keywords(task.query)]
+    scored = []
+    for record in _discovered(repo_root):
+        text = record.path.read_text(encoding="utf-8", errors="replace")
+        low = text.casefold()
+        score = sum(low.count(term) for term in terms)
+        if score:
+            scored.append((score, record.relative_path, text))
+    scored.sort(key=lambda row: (-row[0], row[1]))
+    chosen = scored[:_NATIVE_MAX_FILES] or (
+        [(0, r.relative_path, r.path.read_text(encoding="utf-8", errors="replace")) for r in _discovered(repo_root)[:1]]
+    )
+    parts = []
+    for _, rel, text in chosen:
+        clipped = tok.decode(tok.encode(text)[:_NATIVE_FILE_CAP_TOKENS])
+        parts.append(f"# {rel}\n{clipped}")
+    context = "\n\n".join(parts)
+    sources = [rel for _, rel, _ in chosen]
+    opened = len(chosen)
+    return context, sources, opened + 1, opened, False  # +1 tool call for the grep itself
+
+
+def _broad(repo_root, tok):
+    records = sorted(_discovered(repo_root), key=lambda r: _priority(r.relative_path))
+    parts = []
+    used = 0
+    sources = []
+    for record in records:
+        text = record.path.read_text(encoding="utf-8", errors="replace")
+        chunk = f"# {record.relative_path}\n{text}"
+        cost = tok.count(chunk)
+        if used + cost > _BROAD_CAP_TOKENS:
+            budget = _BROAD_CAP_TOKENS - used
+            parts.append(tok.decode(tok.encode(chunk)[: max(budget, 0)]))
+            sources.append(record.relative_path)
+            break
+        parts.append(chunk)
+        sources.append(record.relative_path)
+        used += cost
+    return "\n\n".join(parts), sources, 1, 1, False
+
+
+def _pasr_full(task, repo_root, budget):
+    request = validate_select_context_request(
+        {
+            "query": task.query,
+            "include": ["."],
+            "budget_tokens": budget,
+            "max_files": 20000,  # real repos have far more than the 100 default
+            "max_file_bytes": 400_000,
+        },
+        workspace_root=repo_root,
+    )
+    return run_select_context(request, write_receipt_file=False)
+
+
+def _pasr(task, repo_root, budget):
+    result = _pasr_full(task, repo_root, budget)
+    sources = [span["source"] for span in result["spans"]] or result["sources"]
+    return result["context"], sources