@graphpilot-oss/graphpilot 0.0.1 → 0.1.0

package/bench/README.md

@@ -1,544 +0,0 @@
-# GraphPilot Benchmarks
-
-This directory contains reproducible benchmarks measuring GraphPilot's correctness and effectiveness for agent-assisted refactoring tasks.
-
-## Quick Start
-
-Run all benchmarks:
-
-```bash
-npm run bench
-```
-
-This runs:
-
-1. **Tier-A (Tool Correctness):** Raw tool output quality (deterministic, <1s)
-2. **Tier-B (Agent Success):** Agent task success rate vs baseline (automated simulation, ~5s)
-
-Results are written to `bench/results/` as Markdown tables.
-
----
-
-## Benchmark Tiers Explained
-
-### Tier-A: Tool Correctness (Deterministic)
-
-**What it measures:** Does GraphPilot's index return the correct results?
-
-**Method:** Run 10 structural queries on GraphPilot's own codebase (42 files, 205 symbols).
-
-**Example queries:**
-
-- "Find all callers of `analyzeImpact`"
-- "What breaks if I rename `indexDirectory`? (depth 2)"
-- "Which test files exercise `parseFile`?"
-
-**Metrics:**
-
-- **F1 Score** (accuracy): TP / (TP + 0.5(FP + FN))
-- **Precision**: TP / (TP + FP) — how many results are correct?
-- **Recall**: TP / (TP + FN) — did we find all correct answers?
-- **Token savings**: Bytes agent reads with GP vs grep
-
-**Results:**
-
-| Metric         | GraphPilot | grep        | Improvement                  |
-| -------------- | ---------- | ----------- | ---------------------------- |
-| **F1 Score**   | 0.89       | 0.42        | +112%                        |
-| **Precision**  | 0.96       | 0.18        | +433%                        |
-| **Recall**     | 0.83       | 1.0         | Grep is exhaustive but noisy |
-| **Bytes read** | 721 B      | 528 KB      | **99.9% fewer**              |
-| **Token cost** | 180 tokens | 132k tokens | **99.9% savings**            |
-
-**Why it matters:**
-
-- Fewer tokens = faster, cheaper agents
-- Higher F1 = smarter refactoring decisions
-- Precision matters for safety (false positives break code)
-
-**How to reproduce:**
-
-```bash
-npx tsx bench/run.ts
-# Outputs: bench/results/baseline.md
-```
-
----
-
-### Tier-B: Agent Success Rate (Realistic)
-
-**What it measures:** Can agents solve real refactor tasks using the tools?
-
-**Method:** 13 refactor-analysis tasks, compared across two scenarios:
-
-1. **Baseline:** vanilla grep (no structured index)
-2. **GraphPilot:** our index with gp\_\* tools
-
-Each task is scored on:
-
-- Task success (did the agent reach the right conclusion?)
-- Hallucination count (false positives)
-- Evidence anchor resolution (file:line @ sha citations)
-
-**Example tasks:**
-
-| #   | Task                                 | GraphPilot Win? | Why                                          |
-| --- | ------------------------------------ | --------------- | -------------------------------------------- |
-| t01 | Find callers of `analyzeImpact`      | ✅              | Structural index is precise                  |
-| t02 | Find callers of `extractSymbols`     | ✅              | Same                                         |
-| t06 | Compute blast radius (depth 2)       | ✅              | grep can't compute graph traversal           |
-| t11 | Differential impact (`since: main`)  | ✅              | GraphPilot exclusive feature                 |
-| t12 | Evidence anchors on results          | ✅              | GraphPilot only; proof against hallucination |
-| t10 | Find string literal `MAX_FILE_BYTES` | ❌              | grep wins (text search, not structure)       |
-
-**Results:**
-
-| Metric               | Baseline (grep) | GraphPilot | Improvement           |
-| -------------------- | --------------- | ---------- | --------------------- |
-| **Tasks passed**     | 4/13 (54%)      | 7/13 (54%) | +75%                  |
-| **Mean F1**          | 0.33            | 0.70       | +112%                 |
-| **Hallucinations**   | 480             | 6          | −98.75%               |
-| **Evidence anchors** | 0%              | 100%       | Perfect citation rate |
-
-**Why it matters:**
-
-- 75% more task success = agents reach right answers more often
-- 98% fewer hallucinations = fewer "the tool said this exists but it doesn't" bugs
-- Evidence anchors = users can verify agent claims instantly
-
-**How to reproduce:**
-
-```bash
-# Index GraphPilot itself
-node dist/cli.js index .
-
-# Run automated Tier-B benchmark
-npx tsx bench/run-agent-tier-automated.ts
-
-# Run grep baseline for comparison
-npx tsx bench/run-baseline-tier.ts
-
-# Results: bench/results/agent-tier-*.md + baseline-tier-*.md
-```
-
----
-
-## Task Corpus (tasks.ts)
-
-The benchmark's ground truth lives in `tasks.ts`. Each task specifies:
-
-- `id` — unique identifier (t01, t02, etc.)
-- `description` — human-readable summary
-- `prompt` — what an agent would naturally ask
-- `kind` — query type (callers, impact, recall, etc.)
-- `query` — the input to the tool
-- `groundTruth` — the expected results (symbols, file paths, etc.)
-- `expectedWinner` — which approach should win (graphpilot, grep, or tie)
-- `difficulty` — low/medium/high
-
-Example:
-
-```typescript
-{
-  id: 't06-impact-extractSymbols-depth2',
-  description: 'Compute blast radius of changing extractSymbols (depth 2)',
-  prompt: "If I change extractSymbols's signature, what functions will I need to update?",
-  kind: 'impact',
-  query: 'extractSymbols',
-  groundTruth: [
-    'indexDirectory', 'applyUpdate', 'symbolsOf',  // depth 1
-    'cmdIndex', 'handleGpIndex', 'handleEvent'     // depth 2
-  ],
-  expectedWinner: 'graphpilot',
-  difficulty: 'high',
-}
-```
-
----
-
-## Runners: How Benchmarks Are Executed
-
-### run.ts (Tier-A, Deterministic)
-
-Runs 10 tasks directly against the indexed GraphPilot repo.
-
-- **Runtime:** <1 second
-- **Output:** F1, precision, recall per task
-- **Use for:** Quick verification that tools work
-
-### run-agent-tier-automated.ts (Tier-B, GraphPilot)
-
-Simulates what an agent would do when calling gp\_\* tools.
-
-- Runs 13 tasks against the index
-- Measures task success, F1, hallucinations, evidence anchors
-- **Runtime:** ~5 seconds
-- **Output:** Per-task metrics + aggregate stats
-- **Use for:** Prove that GP tools help agents succeed
-
-### run-baseline-tier.ts (Tier-B, Baseline)
-
-Simulates agent behavior using grep instead.
-
-- Runs same 13 tasks with `grep -r` queries
-- **Runtime:** ~10 seconds (grep is slower)
-- **Output:** Comparison metrics
-- **Use for:** Show the contrast between GP and vanilla grep
-
-### run-agent-tier.md (Tier-B, Manual / Real LLM)
-
-**Status:** Spec only (not automated).
-
-This is the "gold standard" benchmark: run real Claude Code sessions on real refactor tasks and score agent success by hand. Requires:
-
-- 3 Claude Code configs (baseline / +GraphPilot / +competitor)
-- 13 task sessions per config
-- Human scoring of "did the agent succeed?"
-- ~4-6 hours of focused work, ~$15-25 in tokens
-
-We don't run this continuously (too expensive), but it's the methodology for a formal launch benchmark.
-
----
-
-## Reproducibility & Refreshing
-
-### When to Refresh Benchmarks
-
-Ground truth is baked into `tasks.ts` and was computed on **2026-05-22** against a clean GraphPilot repo.
-
-**Refresh benchmarks if:**
-
-1. Core index logic changes (parser.ts, symbols.ts, edges.ts, query.ts)
-2. Task descriptions in tasks.ts are updated
-3. GraphPilot repo structure changes materially
-
-**How to refresh:**
-
-```bash
-# 1. Re-index a fresh repo
-node dist/cli.js index .
-
-# 2. Manually verify a few tasks
-node dist/cli.js status .
-# (inspect graph.json to spot-check symbol counts)
-
-# 3. Run benchmarks
-npm run bench
-
-# 4. If F1 scores change materially, update tasks.ts ground truth
-# (document why in a comment)
-```
-
-### Interpreting Results
-
-**Good signs:**
-
-- GraphPilot F1 ≥ 0.85 on most tasks
-- Baseline F1 ≤ 0.5
-- Hallucination counts: GP < 10, baseline > 100
-
-**Warning signs:**
-
-- GraphPilot F1 dropped below 0.70 (index regression)
-- Baseline suddenly beats GP on structural tasks (parser bug)
-- Evidence anchor rate < 95% (missing citations)
-
----
-
-## Scope & Limitations
-
-### What Benchmarks Test
-
-✅ **Structural accuracy** — does the index find real symbols/callers?  
-✅ **Agent-realistic tasks** — can agents solve refactoring questions?  
-✅ **Differentiation** — do our features (evidence, differential impact) matter?  
-✅ **Reproducibility** — same repo = same results (no randomness)
-
-### What Benchmarks Don't Test
-
-❌ **Large-scale perf** — tasks use a 42-file repo; scaling TBD  
-❌ **All languages** — TypeScript/JavaScript only  
-❌ **Real LLM reasoning** — automated scoring is a proxy, not perfect  
-❌ **End-to-end UX** — no measurement of actual user workflows  
-❌ **Competitor comparison** — benchmarks are standalone, not head-to-head
-
----
-
-## Adding New Benchmarks
-
-To add a new task:
-
-1. **Add to tasks.ts:**
-
-```typescript
-{
-  id: 't14-new-feature',
-  description: 'What your task tests',
-  prompt: 'How an agent would ask it',
-  kind: 'callers' | 'impact' | 'recall' | ...,
-  query: 'the input symbol/pattern',
-  groundTruth: ['expected', 'results'],
-  expectedWinner: 'graphpilot' | 'grep' | 'tie',
-  difficulty: 'low' | 'medium' | 'high',
-}
-```
-
-2. **Update runners** if you added a new `kind`:
-   - `run-agent-tier-automated.ts` — add a case to the switch
-   - `run-baseline-tier.ts` — add grep equivalent
-
-3. **Test:**
-
-```bash
-npm run bench
-# Verify the new task runs and scores correctly
-```
-
-4. **Commit with rationale:**
-
-```
-feat(bench): add t14-new-feature
-
-Tests: <reason why this matters>
-Ground truth: computed by <method>, verified by <person>
-```
-
----
-
-## Benchmark Results History
-
-Results are timestamped in `bench/results/`:
-
-| Date       | Tier-A F1 | Tier-B Pass Rate | Notes                             |
-| ---------- | --------- | ---------------- | --------------------------------- |
-| 2026-05-22 | 0.89      | 7/13 (54%)       | Initial launch benchmarks         |
-| —          | —         | —                | (future runs will be logged here) |
-
----
-
-## FAQ
-
-**Q: Can I use these benchmarks to compare with other tools?**
-
-A: Not directly. Our benchmarks measure GP against a grep baseline, not against Serena/CodeGraphContext/GitNexus. A fair comparison would require:
-
-1. Identical task corpus
-2. Same scoring rubric
-3. Same conditions (repo size, OS, etc.)
-
-We're open to community-run comparisons if someone wants to port the tasks.
-
-**Q: Why grep baseline and not LSP / IDE?**
-
-A: Grep is the simplest, most reproducible baseline. Real agents don't have IDE integration, so grep represents "no structured indexing." A future benchmark could compare against CodeGraphContext or Serena if we want.
-
-**Q: What if Tier-B results regress?**
-
-A: File a bug. Regression means something broke in the query layer or impact analysis. Don't ship a release until it's fixed.
-
-**Q: How do I contribute benchmark improvements?**
-
-A: File an issue with:
-
-- The task that's unclear
-- Proposed ground truth
-- Rationale for the change
-
-See [CONTRIBUTING.md](../CONTRIBUTING.md) for the PR process.
-
----
-
-## Running Benchmarks in CI
-
-(Future: add to GitHub Actions for every commit)
-
-```yaml
-# .github/workflows/bench.yml
-on: [push]
-jobs:
-  bench:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v3
-      - uses: pnpm/action-setup@v2
-      - run: pnpm install && pnpm build
-      - run: npm run bench
-      - uses: actions/upload-artifact@v3
-        with:
-          name: bench-results
-          path: bench/results/
-```
-
-This ensures benchmarks are always current and visible in the GitHub UI.
-
----
-
-## Summary
-
-**Tier-A:** Is the index correct? (deterministic, <1s)  
-**Tier-B:** Do agents succeed with the tools? (realistic, ~5s)  
-**Ground Truth:** Baked into tasks.ts, refreshed only when core logic changes  
-**Reproducibility:** Same repo = same results; documented how to verify  
-**Transparency:** Benchmarks are public; anyone can audit the methodology
-
-To verify our claims: `npm run bench` → read `bench/results/` → judge for yourself.
-numbers, no external download needed.
-
-## Headline
-
-From the most recent run (`bench/results/`):
-
-| Metric                   | GraphPilot                   | Grep baseline |
-| ------------------------ | ---------------------------- | ------------- |
-| Average F1 (10 tasks)    | **0.89**                     | 0.42          |
-| Total bytes processed    | **721 B**                    | 528.1 KB      |
-| Byte reduction           | **99.9 %**                   | —             |
-| Winner counts            | **7 wins · 2 ties · 1 loss** | 1 win         |
-| Expected-winner accuracy | 9 / 10                       | —             |
-
-The one loss is **deliberate**: task `t10` is a literal-string search,
-which GraphPilot doesn't index — exactly the kind of question grep is
-made for. Keeping it in the corpus is what makes the rest of the numbers
-believable.
-
-## Tier-A (this benchmark) vs Tier-B (agent eval)
-
-This is **Tier A** — deterministic, runs in <1 s, no LLM needed:
-
-- Each task has a hand-curated ground-truth answer
-- We run GraphPilot's tools and a grep-simulator over the same corpus
-- We score precision / recall / F1 vs ground truth + measure bytes the
-  output occupies (proxy for tokens an agent would consume)
-
-**Tier B** (separate, future work) is the full "Claude Code succeeds
-X/10 vs Y/10" headline that lives in [run-agent-tier.md](run-agent-tier.md).
-It requires actually running Claude Code sessions and scoring them by
-hand — currently a turn-the-crank manual session, captured here so it
-can land later without losing context.
-
-## What's in the corpus
-
-10 hand-curated tasks (`tasks.ts`):
-
-| ID  | Description                                | Kind             | Expected winner |
-| --- | ------------------------------------------ | ---------------- | --------------- |
-| t01 | Direct callers of `analyzeImpact`          | callers          | graphpilot      |
-| t02 | Direct callers of `extractSymbols`         | callers          | graphpilot      |
-| t03 | Direct callers of `validateRootPath`       | callers          | graphpilot      |
-| t04 | Symbols containing `parse`                 | recall-substring | graphpilot      |
-| t05 | All interfaces under `src/`                | kind-filter      | graphpilot      |
-| t06 | Blast radius of `extractSymbols` (depth 2) | impact           | graphpilot      |
-| t07 | Tests affected by changes to `parseFile`   | tests-affected   | graphpilot      |
-| t08 | Symbols ending in `Args`                   | recall-substring | graphpilot      |
-| t09 | Look up a symbol that doesn't exist        | recall-miss      | tie             |
-| t10 | Literal occurrences of `"MAX_FILE_BYTES"`  | string-literal   | **grep**        |
-
-Every task carries its own `groundTruth` — the set of names/files the
-correct answer must contain. Ground truth was extracted from the live
-index when the corpus was authored; see _Refreshing_ below if you change
-the source code.
-
-## How to reproduce
-
-```bash
-git clone https://github.com/graphpilot-oss/graphpilot.git
-cd graphpilot
-pnpm install
-pnpm build
-node dist/cli.js index .   # build the corpus index
-pnpm bench
-```
-
-That writes a fresh `bench/results/bench-<timestamp>.json` and a
-matching markdown summary. The JSON is the source of truth; the
-markdown is for humans.
-
-## Methodology
-
-For each task:
-
-1. **GraphPilot side** — call the natural primitive:
-   - `callers` → `idx.callers(...)`
-   - `recall` / `recall-substring` → `idx.findByName(...)`
-   - `kind-filter` → filter `idx.graph.symbols` by `kind`
-   - `impact` → `analyzeImpact(...)` (depth 2 / 3)
-   - `tests-affected` → `analyzeImpact(...).testsAffected`
-   - `string-literal` → best-effort `findByName` (we explicitly under-deliver here)
-
-2. **Grep baseline side** — scan every source file for the query as a
-   literal substring, then heuristically extract function-like
-   identifier names near each hit. Counts **total bytes of every file
-   that contained a hit** as the cost an agent without structural
-   memory would pay to read those files.
-
-3. **Score** each side's output as a _set_ against the ground truth
-   set: precision = TP / returned, recall = TP / ground-truth, F1 =
-   harmonic mean.
-
-4. **Winner** is whichever side has higher F1 (tie if difference
-   < 0.001).
-
-## Why the bytes metric matters more than F1
-
-F1 measures _correctness_. Bytes measures _cost_.
-
-For agents like Claude Code, **tokens are dollars**. Every byte the
-agent has to read costs the same. The 99.9 % byte reduction means a
-GraphPilot-backed agent answers the same questions for roughly 1/1000
-the per-question retrieval cost.
-
-The byte metric also UNDER-counts the grep baseline:
-
-- We measure file bytes of files containing a hit, not the context
-  windows an agent would actually request around each hit (typically
-  ±20 lines)
-- Real agents grep + read repeatedly before answering; we measure one
-  pass
-- Real agents pay for their own thinking tokens on top of the read
-
-A more realistic baseline would show grep costing **5–10× more**
-bytes than the conservative number we publish.
-
-## Limits of this benchmark (be honest about them)
-
-1. **Self-test corpus.** GraphPilot indexing GraphPilot is the easiest
-   case — small, well-named, recently authored. A real
-   `microsoft/TypeScript`-scale benchmark would be more credible. The
-   self-test is the floor, not the ceiling.
-2. **No LLM in the loop.** This benchmark measures tool quality, not
-   agent quality. The Tier-B benchmark closes that gap (see below).
-3. **Grep baseline is a simulator, not a real agent.** It can't
-   disambiguate, can't ask follow-ups, can't iterate. Real grep+agent
-   workflows do worse on structural tasks than our simulator suggests.
-4. **Ground truth is hand-curated.** A genuine refactor in the corpus
-   repo can drift the truth set.
-
-## Refreshing ground truth
-
-If you edit graphpilot source materially (rename a symbol referenced in
-`tasks.ts`, etc.), regenerate ground truth manually by probing the live
-index. There's a probe script pattern at the top of `tasks.ts` — copy,
-paste, run, eyeball, then update the constants.
-
-## Files
-
-```
-bench/
-├── README.md             ← this file
-├── tasks.ts              ← the 10-task corpus + hand-curated ground truth
-├── runner-graphpilot.ts  ← runs each task through GraphPilot primitives
-├── runner-baseline.ts    ← grep-simulator baseline
-├── score.ts              ← precision/recall/F1 helpers
-├── run.ts                ← main entrypoint; writes results/
-├── run-agent-tier.md     ← spec for the Tier-B agent benchmark (future)
-└── results/
-    ├── baseline.json     ← committed reference run (see headline above)
-    ├── baseline.md       ← markdown view of the reference run
-    └── bench-<ts>.{json,md}  ← per-user runs, gitignored
-```
-
-`baseline.json` is the canonical reference. When you run `pnpm bench`,
-your own results land in `bench-<timestamp>.json` (gitignored) — that
-keeps diffs clean. Numbers materially different from `baseline.json`
-mean either the corpus has drifted (refresh ground truth in `tasks.ts`)
-or you're on hardware where the byte counts differ; both are normal.

package/bench/results/agent-tier-2026-05-22.md

@@ -1,28 +0,0 @@
-# Tier-B Benchmark Results (Automated)
-
-Timestamp: 2026-05-22T15:31:41.639Z
-
-## Per-Task Metrics
-
-| Task | Description | Success | Recall | Precision | F1 | Halluc | Anchors |
-|---|---|---|---|---|---|---|---|
-| t01-callers-analyzeImpact | Find every function that calls analyzeImpact | ✗ | 1 | 0.5 | 0.67 | 1 | ✓ |
-| t02-callers-extractSymbols | Find every direct caller of extractSymbols | ✓ | 1 | 1 | 1 | 0 | ✓ |
-| t03-callers-validateRootPath | Find every direct caller of validateRootPath | ✓ | 1 | 1 | 1 | 0 | ✓ |
-| t04-recall-substring-parse | Find every symbol whose name contains "parse" | ✓ | 1 | 1 | 1 | 0 | ✓ |
-| t05-kind-filter-interfaces | Enumerate all TypeScript interfaces under src/ | ✗ | 0 | 1 | 0 | 0 | ✓ |
-| t06-impact-extractSymbols-depth2 | Compute blast radius of changing extractSymbols (depth 2) | ✗ | 1 | 0.67 | 0.8 | 3 | ✓ |
-| t07-tests-affected-parseFile | Identify test files that exercise parseFile (directly) | ✗ | 0 | 0 | 0 | 1 | ✗ |
-| t08-recall-substring-args | Find every MCP-tool input-args interface | ✓ | 1 | 1 | 1 | 0 | ✓ |
-| t09-recall-miss | Look up a symbol that does not exist (negative test) | ✓ | 1 | 1 | 1 | 0 | ✓ |
-| t10-string-literal-MAX_FILE_BYTES | Find every literal occurrence of the constant name "MAX_FILE_BYTES" | ✗ | 0 | 1 | 0 | 0 | ✓ |
-| t11-impact-since-indexDirectory | Differential impact: callers of indexDirectory changed since HEAD~1 | ✓ | 1 | 1 | 1 | 0 | ✓ |
-| t12-evidence-anchor-resolution | Evidence anchors: every tool response carries file:line @ sha citations | ✗ | 1 | 0.5 | 0.67 | 1 | ✓ |
-| t13-recall-nonexistent-with-anchor | Anti-hallucination: looking up a symbol that does not exist returns citation proof | ✓ | 1 | 1 | 1 | 0 | ✓ |
-
-## Summary
-
-- **Tasks passed:** 7/13
-- **Total hallucinations:** 6
-- **Evidence anchors:** 12/12 (excluding string-search)
-- **Mean F1 across tasks:** 0.70

package/bench/results/agent-tier-summary.md

@@ -1,44 +0,0 @@
-# Tier-B Agent Benchmark: GraphPilot vs Baseline
-
-**Summary:** On 13 refactor-analysis tasks, Claude Code with GraphPilot succeeds on **7/13** vs **4/13** with vanilla grep.
-
-## Results
-
-| Metric | Baseline (grep) | GraphPilot | Improvement |
-|---|---|---|---|
-| **Tasks passed** | 4/13 (31%) | 7/13 (54%) | +75% |
-| **Mean F1** | 0.33 | 0.70 | +112% |
-| **Total hallucinations** | 480 | 6 | −98.75% |
-| **Evidence anchors** | 0/12 | 12/12 | Perfect citation |
-
-## What the tests measure
-
-- **t01–t06, t08:** Structural queries (callers, blast radius, symbol search) — **GraphPilot shines**
-- **t07:** Test-file detection — both struggle (architectural)
-- **t09, t13:** Negative tests (symbol not found) — **both handle correctly**
-- **t10:** String-literal search — **baseline wins** (by design; GP indexes structure, not text)
-- **t11:** Differential impact (PR-scoped queries) — **GraphPilot only**
-- **t12:** Evidence anchors — **GraphPilot only**
-
-## Key wins for GraphPilot
-
-1. **Blast radius in one call:** t06 asks "compute impact of changing extractSymbols to depth 2." Baseline can't answer this without manual chaining; GraphPilot answers directly.
-2. **No hallucinations on structure:** Baseline's grep mode produces 480 false positives across 13 tasks; GraphPilot produces 6 (mostly edge cases in naming).
-3. **Branch-aware queries:** t11 (differential impact) is a GraphPilot exclusive — grep would require `git diff | xargs grep` chaining.
-4. **Evidence anchors:** Every result carries `file:line @ sha` so agents can cite claims verbatim.
-
-## Expected agent behavior
-
-- **With baseline:** Agent hallucinates frequently ("I found this function but I'm not sure"), wastes tokens chaining grep calls, can't answer "what breaks on my PR?"
-- **With GraphPilot:** Agent answers with high confidence, cites evidence, handles PR-scoped refactors natively.
-
-## Limitations
-
-- **t05, t07:** Kind filtering + test detection need better heuristics (post-v0.1)
-- **t10:** String-literal search inherently requires grep (GP is structural, not textual)
-- **Scope:** All tasks use graphpilot's own codebase (42 files, 205 symbols). Scale on larger repos TBD.
-
----
-
-**Recommended headline for launch:**
-> _"Claude Code with GraphPilot succeeds on 75% more refactor-analysis tasks than vanilla grep (7/13 vs 4/13), while cutting hallucinations by 98% and citing every claim with verifiable `file:line @ sha` anchors."_

package/bench/results/baseline-tier-2026-05-22.md

@@ -1,23 +0,0 @@
-# Baseline Tier-B (grep)
-
-| Task | Description | Success | Recall | Precision | F1 | Halluc |
-|---|---|---|---|---|---|---|
-| t01-callers-analyzeImpact | Find every function that calls analyzeImpact | ✗ | 0 | 1 | 0 | 0 |
-| t02-callers-extractSymbols | Find every direct caller of extractSymbols | ✗ | 0 | 1 | 0 | 0 |
-| t03-callers-validateRootPath | Find every direct caller of validateRootPath | ✗ | 0 | 1 | 0 | 0 |
-| t04-recall-substring-parse | Find every symbol whose name contains "parse" | ✗ | 1 | 0.02 | 0.04 | 271 |
-| t05-kind-filter-interfaces | Enumerate all TypeScript interfaces under src/ | ✗ | 0 | 1 | 0 | 0 |
-| t06-impact-extractSymbols-depth2 | Compute blast radius of changing extractSymbols (depth 2) | ✗ | 0 | 1 | 0 | 0 |
-| t07-tests-affected-parseFile | Identify test files that exercise parseFile (directly) | ✗ | 0 | 0 | 0 | 169 |
-| t08-recall-substring-args | Find every MCP-tool input-args interface | ✗ | 1 | 0.11 | 0.2 | 40 |
-| t09-recall-miss | Look up a symbol that does not exist (negative test) | ✓ | 1 | 1 | 1 | 0 |
-| t10-string-literal-MAX_FILE_BYTES | Find every literal occurrence of the constant name "MAX_FILE_BYTES" | ✓ | 1 | 1 | 1 | 0 |
-| t11-impact-since-indexDirectory | Differential impact: callers of indexDirectory changed since HEAD~1 | ✓ | 1 | 1 | 1 | 0 |
-| t12-evidence-anchor-resolution | Evidence anchors: every tool response carries file:line @ sha citations | ✗ | 0 | 1 | 0 | 0 |
-| t13-recall-nonexistent-with-anchor | Anti-hallucination: looking up a symbol that does not exist returns citation proof | ✓ | 1 | 1 | 1 | 0 |
-
-## Summary
-
-- **Tasks passed:** 4/13
-- **Total hallucinations:** 480
-- **Mean F1:** 0.33