codebase-index 1.7.0__tar.gz → 1.9.0__tar.gz

codebase_index-1.9.0/.claude/skills/codebase-index/.skill_version

@@ -0,0 +1 @@
+1.9.0

{codebase_index-1.7.0 → codebase_index-1.9.0}/.claude-plugin/plugin.json

@@ -3,7 +3,7 @@
   "name": "codebase-index",
   "displayName": "Codebase Index",
   "description": "Give Claude a precise local map of your codebase: find implementations, trace behavior, and predict change impact with file-line evidence.",
-  "version": "1.7.0",
+  "version": "1.9.0",
   "author": {
     "name": "codebase-index contributors"
   },

codebase_index-1.9.0/.codex/skills/codebase-index/.skill_version

@@ -0,0 +1 @@
+1.9.0

codebase_index-1.9.0/.opencode/skills/codebase-index/.skill_version

@@ -0,0 +1 @@
+1.9.0

{codebase_index-1.7.0 → codebase_index-1.9.0}/CHANGELOG.md

@@ -6,6 +6,109 @@ All notable changes to this project are documented here. The format is based on
 
 ## [Unreleased]
 
+## [1.9.0] - 2026-09-02
+
+### Added
+
+- **Objective ground truth from git history.** `tests/eval/gen_queries.py` mints a
+  retrieval benchmark from any git repository by pairing a human-written commit
+  subject with the files that commit actually changed. Unlike docstring-derived
+  benchmarks the query text is not copied into the document being retrieved, so the
+  set is leak-free by construction; merges, reverts, releases, version bumps,
+  sweeping refactors, changelog-style answers and benchmark scaffolding are all
+  filtered out.
+- **Multi-corpus, multi-language evaluation.** `run_eval.py --corpus REPO:QUERIES`
+  pools several repositories into one benchmark so ranking changes are validated
+  outside this repository and outside Python. The 1.9.0 defaults were measured on
+  305 queries across Python, Java and TypeScript corpora.
+- **Significance testing.** Every non-baseline row now reports a paired bootstrap
+  95% confidence interval and a paired permutation p-value (both seeded, therefore
+  reproducible). Query sets of this size have a noise floor of several MRR points;
+  signals now ship on the strength of that test rather than the sign of a delta.
+- **Context-noise metrics.** The report adds mean emitted snippet tokens, duplicate
+  rate of returned results, and p99 latency alongside the existing IR metrics.
+
+### Changed
+
+- **Fusion now scores cross-retriever agreement.** RRF fuses on
+  `(path, line-bucket)`, so a symbol definition at line 40 and a lexical hit at line
+  120 in the same file fused as two unrelated candidates — two retrievers agreeing
+  on a file produced two weak results instead of one strong one, and cross-source
+  agreement never reached the score. Each candidate now also receives, at
+  `file_agreement_weight` (0.4), the RRF mass of every retriever that found its file
+  at another locator, excluding retrievers already counted at that locator.
+  Ablatable via `RetrievalTuning(file_agreement=False)`.
+- **Documentation demotion deepened** from -0.05 to -0.20. Prose describing a
+  feature matches a natural-language question more literally than the code
+  implementing it, so design notes and plans were displacing the modules they
+  describe. Because this only reorders prose relative to code, documentation-seeking
+  queries improved as well (category MRR 0.579 → 0.612). Generated/vendor paths move
+  to -0.25 so they remain the least-preferred role, and `MAX_ABS_PRIOR` now pins the
+  invariant that priors stay tiebreakers.
+- **Fuzzy identifier matching is now a recall fallback.** It runs only when the
+  precise symbol lookup named no symbol and returned fewer than
+  `fuzzy_fallback_min` (3) rows. It moved no ranking metric across 305 queries while
+  accounting for ~20% of query latency; typo and acronym recall is unchanged because
+  those are exactly the queries where the precise lookup comes up empty.
+- **Candidate over-fetch is explicit.** `candidate_pool_multiplier` (default 2)
+  replaces the implicit widening that happened whenever dedup or MMR was enabled.
+  Making it explicit revealed that the quality previously credited to SimHash dedup
+  was really the wider pool; dedup is retained for what it does measurably do, which
+  is cutting the duplicate rate of returned snippets from ~1.6% to ~0%.
+
+### Fixed
+
+- **Synonym matches were reported as exact symbol matches.** `is_exact` came from
+  SQL and was relative to whichever needle retrieved the row, so a synonym
+  expansion ("config" for "configuration") marked an unrelated symbol as an exact
+  match — worth a +0.20 rerank bonus and an unconditional `high` confidence.
+  Exactness is now judged against the terms the user actually typed.
+- **Duplicate suppression was order-dependent on ties.** Equal-scoring duplicates
+  handed the slot to whichever copy arrived last, contradicting the documented
+  "ties favor input order" and making the retained snippet depend on retriever
+  emission order.
+
+### Performance
+
+- Query latency roughly halves: p50 78.6 ms → 51.2 ms, p95 193.2 ms → 94.9 ms,
+  p99 277.1 ms → 152.2 ms on the pooled three-repository benchmark, from the fuzzy
+  fallback plus a SimHash fingerprint that folds repeated tokens by multiplicity and
+  caches token digests. Fingerprint output is bit-for-bit unchanged.
+- Mean emitted snippet tokens fall 1183 → 1091 per query.
+
+### Retrieval quality
+
+Pooled over 305 queries (Python, Java, TypeScript), v1.8.0 → 1.9.0:
+
+| Metric | v1.8.0 | 1.9.0 | Δ | p |
+|---|---|---|---|---|
+| MRR | 0.564 | 0.591 | +0.027 | <0.001 |
+| MAP | 0.433 | 0.461 | +0.028 | <0.001 |
+| nDCG@10 | 0.503 | 0.527 | +0.024 | <0.001 |
+| recall@5 | 0.529 | 0.560 | +0.031 | <0.001 |
+| P@5 | 0.182 | 0.192 | +0.009 | 0.001 |
+| hit@3 | 0.649 | 0.666 | +0.016 | 0.124 |
+
+## [1.8.0] - 2026-09-02
+
+### Added
+
+- **Reproducible retrieval evaluation.** Added a self-repository ground-truth
+  query suite with Recall@K, MRR, nDCG, hit rate, precision, MAP, useful-context,
+  latency percentiles, and one-signal ablations.
+
+### Changed
+
+- **Hybrid retrieval quality.** Natural-language lexical queries now use safe,
+  down-weighted identifier/synonym expansion and soft term coverage; exact symbol
+  lookup preserves framing-word tolerance and bounded fuzzy matching.
+- **Packaging compatibility.** Cap the build backend below the Metadata 2.5
+  default until the release validation toolchain supports that metadata version.
+- **Ranking defaults are evidence-driven.** Implementation/test/documentation
+  source priors are calibrated from the benchmark. Graph propagation is bounded
+  and intent-directed; graph and MMR signals remain opt-in because ablations
+  reduced direct retrieval quality on the reproducible corpus.
+
 ## [1.7.0] - 2026-07-29
 
 ### Added
@@ -401,7 +504,8 @@ All notable changes to this project are documented here. The format is based on
 - Hooks example + `watch` mode for keeping the index fresh without blocking the edit loop (M8).
 - `doctor`, `stats`, `clean` diagnostics/maintenance commands.
 
-[Unreleased]: https://github.com/denfry/codebase-index/compare/v1.7.0...HEAD
+[Unreleased]: https://github.com/denfry/codebase-index/compare/v1.8.0...HEAD
+[1.8.0]: https://github.com/denfry/codebase-index/compare/v1.7.0...v1.8.0
 [1.7.0]: https://github.com/denfry/codebase-index/compare/v1.6.0...v1.7.0
 [1.6.0]: https://github.com/denfry/codebase-index/compare/v1.5.0...v1.6.0
 [1.5.0]: https://github.com/denfry/codebase-index/compare/v1.4.0...v1.5.0

{codebase_index-1.7.0 → codebase_index-1.9.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: codebase-index
-Version: 1.7.0
+Version: 1.9.0
 Summary: A precise local codebase map for AI coding agents: find, trace, and predict with file-line evidence.
 Project-URL: Homepage, https://github.com/denfry/codebase-index
 Project-URL: Documentation, https://github.com/denfry/codebase-index/tree/main/docs
@@ -321,7 +321,7 @@ See [MCP.md](docs/MCP.md) for client configuration.
 
 ## Project status
 
-The latest released line is **1.7.0**. It includes:
+The latest released line is **1.9.0**. It includes:
 
 - hybrid and optional vector retrieval;
 - Tree-sitter symbol extraction across the documented language tiers;
@@ -329,14 +329,22 @@ The latest released line is **1.7.0**. It includes:
 - architecture communities, central nodes, and surprising cross-module links;
 - shortest dependency paths and node descriptions;
 - token-budgeted and skeletonized retrieval packets;
+- benchmark-calibrated lexical expansion, fuzzy identifier matching, and source-aware ranking;
+- rank fusion that scores cross-retriever agreement at file level, not just at a locator;
+- bounded, intent-directed graph discovery with optional diversity and duplicate suppression;
 - CLI, Skill, plugin, and MCP delivery;
 - incremental updates, watch hooks, diagnostics, skill rollback, and diff-aware
-  impact analysis.
-
-Planned work is deliberately separated from shipped capability. The next
-product priorities are stronger real-repository evaluations, typed framework
-edges, and an even more direct task-context workflow. See the
-[roadmap](docs/ROADMAP.md).
+  impact analysis;
+- a multi-repository retrieval evaluation with leak-free git-derived ground truth,
+  one-signal ablations, and paired significance tests
+  ([tests/eval](tests/eval/README.md)).
+
+Every shipped ranking signal has to survive that evaluation: 1.9.0 removed the
+cost of two signals that could not demonstrate a benefit and rejected several
+plausible ones outright (IDF-weighted coverage, stemming, graph propagation, MMR,
+a file-length prior). Planned work is deliberately separated from shipped
+capability. The next product priorities are typed framework edges and an even more
+direct task-context workflow. See the [roadmap](docs/ROADMAP.md).
 
 ## Documentation
 

{codebase_index-1.7.0 → codebase_index-1.9.0}/README.md

@@ -271,7 +271,7 @@ See [MCP.md](docs/MCP.md) for client configuration.
 
 ## Project status
 
-The latest released line is **1.7.0**. It includes:
+The latest released line is **1.9.0**. It includes:
 
 - hybrid and optional vector retrieval;
 - Tree-sitter symbol extraction across the documented language tiers;
@@ -279,14 +279,22 @@ The latest released line is **1.7.0**. It includes:
 - architecture communities, central nodes, and surprising cross-module links;
 - shortest dependency paths and node descriptions;
 - token-budgeted and skeletonized retrieval packets;
+- benchmark-calibrated lexical expansion, fuzzy identifier matching, and source-aware ranking;
+- rank fusion that scores cross-retriever agreement at file level, not just at a locator;
+- bounded, intent-directed graph discovery with optional diversity and duplicate suppression;
 - CLI, Skill, plugin, and MCP delivery;
 - incremental updates, watch hooks, diagnostics, skill rollback, and diff-aware
-  impact analysis.
-
-Planned work is deliberately separated from shipped capability. The next
-product priorities are stronger real-repository evaluations, typed framework
-edges, and an even more direct task-context workflow. See the
-[roadmap](docs/ROADMAP.md).
+  impact analysis;
+- a multi-repository retrieval evaluation with leak-free git-derived ground truth,
+  one-signal ablations, and paired significance tests
+  ([tests/eval](tests/eval/README.md)).
+
+Every shipped ranking signal has to survive that evaluation: 1.9.0 removed the
+cost of two signals that could not demonstrate a benefit and rejected several
+plausible ones outright (IDF-weighted coverage, stemming, graph propagation, MMR,
+a file-length prior). Planned work is deliberately separated from shipped
+capability. The next product priorities are typed framework edges and an even more
+direct task-context workflow. See the [roadmap](docs/ROADMAP.md).
 
 ## Documentation
 

{codebase_index-1.7.0 → codebase_index-1.9.0}/docs/ARCHITECTURE.md

@@ -2,7 +2,7 @@
 
 ## 1. Overview
 
-`codebase-index` is a **local-first** code intelligence layer for AI coding agents. In `1.7.0`
+`codebase-index` is a **local-first** code intelligence layer for AI coding agents. In `1.9.0`
 it has two shipped faces:
 
 1. **A Claude Code Skill** (`.claude/skills/codebase-index/SKILL.md`) that Claude auto-invokes for
@@ -97,8 +97,9 @@ codebase-index/
     ├── graph/                   # builder.py (edge resolution), expand.py (impact),
     │                            #   export.py (HTML graph)
     ├── storage/                 # db.py (pragmas, schema, version guard), schema.sql, repo.py
-    ├── retrieval/               # intent.py, searchers.py, fusion.py, rerank.py,
-    │                            #   budget.py, pipeline.py, types.py
+    ├── retrieval/               # intent.py, searchers.py, fusion.py, rerank.py, priors.py,
+    │                            #   lexical.py, fuzzy.py, diversity.py, skeleton.py,
+    │                            #   tuning.py (ablation contract), budget.py, pipeline.py, types.py
     ├── embeddings/              # backend.py, noop.py (default), local.py, external.py — opt-in
     ├── output/                  # markdown.py, json.py, redact.py
     ├── watch/                   # watcher.py (optional, watchdog-based)

{codebase_index-1.7.0 → codebase_index-1.9.0}/docs/BENCHMARKS.md

@@ -1,10 +1,11 @@
 # Benchmarks
 
-`codebase-index` has three benchmark surfaces. Read them with their status in
+`codebase-index` has four benchmark surfaces. Read them with their status in
 mind — the whole point of this page is to keep evidence and aspiration separate.
 
 | Surface | What it is | Status | Use it as |
 |---|---|---|---|
+| Retrieval eval (`tests/eval/`) | Ranking quality vs a fixed baseline across multiple real repositories, with significance tests | **Proven (relative)** | The gate for ranking changes; measures *deltas*, not absolute superiority |
 | Public suite (`tests/benchmark_public.py`) | Deterministic synthetic multi-language fixture with the full metric framework | **Toy/synthetic** | CI regression gate + metric shape, **not** product-quality evidence |
 | Smoke/perf (`test_perf_smoke.py`, `test_benchmark_comparison.py`) | Latency + output-size guards on a tiny fixture | **Toy/smoke** | Regression checks only |
 | Honest real-repo (`tests/benchmark_honest.py`) | 55k LOC Java repo, recall@3 vs disciplined `rg` baseline, symmetric token accounting | **Proven (one repo)** | The only headline product-quality number we stand behind today |
@@ -15,15 +16,33 @@ Do not write, imply, or ship any of these until a run with published logs exists
 
 - Any 10k / 100k / 1M LOC scale or speed claim (no real run at that size).
 - "Beats Cursor / Sourcegraph / Codebase-Memory MCP" — no head-to-head exists.
-- Per-language quality claims beyond Java (the honest run is Java-only).
+- Per-language *absolute* quality claims beyond Java. The retrieval eval covers
+  Python, Java and TypeScript, but it measures this system against its own earlier
+  versions — it says a change helped, not that the product beats an alternative.
 - Generic "Nx faster" / "Nx fewer tokens" without naming the baseline and repo.
-- Latency claims — the honest run explicitly does not headline latency
-  (Python process start dominates; real `rg` is tens of ms).
+- Latency claims against external tools — the honest run explicitly does not
+  headline latency (Python process start dominates; real `rg` is tens of ms).
+  Version-over-version latency from the retrieval eval is in-process and is only
+  comparable to other runs of that harness.
 
 The defensible headline today is exactly: **on one 55k LOC Java repo, recall@3 was
 70% (index) vs 40% (`rg`+window), using ~13× fewer answer tokens.** Everything
 else is roadmap.
 
+## Retrieval evaluation (ranking gate)
+
+See [tests/eval/README.md](../tests/eval/README.md) for the full protocol. In short:
+ground truth comes from hand-written queries verified against the source tree and
+from commit-subject → changed-files pairs mined from git history (leak-free: the
+query text is not in the indexed corpus). Corpora are pooled across languages, one
+index per corpus is shared by every variant, and every non-baseline row carries a
+paired bootstrap CI and permutation p-value.
+
+`1.9.0` was measured over 305 queries across Python, Java and TypeScript corpora
+against `1.8.0`: MRR +0.027, MAP +0.028, nDCG@10 +0.024, recall@5 +0.031 (all
+p < 0.001), with p50 latency 78.6 ms → 51.2 ms. These are version-over-version
+ranking deltas on those corpora, not a universal quality claim.
+
 ## Public benchmark suite
 
 Run:

{codebase_index-1.7.0 → codebase_index-1.9.0}/docs/FAQ.md

@@ -14,7 +14,7 @@ pip install codebase-index        # or: pipx install codebase-index
 ```
 
 To pin an exact version or grab an unreleased commit, install from a GitHub tag
-instead: `pip install "codebase-index @ git+https://github.com/denfry/codebase-index.git@v1.7.0"`.
+instead: `pip install "codebase-index @ git+https://github.com/denfry/codebase-index.git@v1.8.0"`.
 
 Then run `codebase-index init` inside your project and `codebase-index index` to build
 the first index. In Claude Code you can instead install the plugin
@@ -162,11 +162,12 @@ Yes. Use any of these methods:
 
 ## Is it production-ready?
 
-Yes — `codebase-index` is released as **v1.7.0**. The core indexing and search
-functionality is implemented and tested. The current `1.7.0` package includes:
+Yes — `codebase-index` is released as **v1.8.0**. The core indexing and search
+functionality is implemented and tested. The current `1.8.0` package includes:
 
-- Hybrid FTS/path/symbol/vector retrieval
-- Import/call/reference graph expansion and `impact`
+- Hybrid FTS/path/symbol/vector retrieval with benchmark-calibrated lexical expansion
+  and bounded fuzzy identifier matching
+- Import/call/reference graph expansion, intent-directed graph discovery, and `impact`
 - Diff-aware blast-radius analysis for tracked working-tree changes
 - Optional local embeddings, with external embeddings gated behind explicit opt-in
 - Hooks and watch mode for freshness

{codebase_index-1.7.0 → codebase_index-1.9.0}/docs/INSTALLATION.md

@@ -67,7 +67,7 @@ pipx install codebase-index            # isolated environment
 uv tool install codebase-index         # uv-managed tool
 
 # Pin to a GitHub tag for an exact or unreleased version
-pip install "codebase-index @ git+https://github.com/denfry/codebase-index.git@v1.7.0"
+pip install "codebase-index @ git+https://github.com/denfry/codebase-index.git@v1.8.0"
 
 # From source (editable mode)
 git clone https://github.com/denfry/codebase-index.git
@@ -90,7 +90,7 @@ pip install -e ".[embeddings-local,watch,dev]"
 
 ### uvx / Homebrew status
 
-As of `1.7.0`, **PyPI is shipped** — `pip install codebase-index` and
+As of `1.8.0`, **PyPI is shipped** — `pip install codebase-index` and
 `pipx install codebase-index` are the verified paths. `uvx codebase-index init`,
 Homebrew tap installation, signed checksums, and SBOMs remain distribution
 targets for a more complete release story.
@@ -130,7 +130,7 @@ Expected output:
 === codebase-index Doctor ===
 
 [OK] Python 3.12 (requires 3.11+)
-[OK] codebase-index package installed (v1.7.0)
+[OK] codebase-index package installed (v1.8.0)
 [OK] tree-sitter is available
 [INFO] Cache directory not yet created: ...
 [INFO] Skill not installed in .claude/skills/

codebase_index-1.9.0/docs/RETRIEVAL.md

@@ -0,0 +1,229 @@
+# Retrieval Pipeline
+
+The retrieval engine turns a natural-language or symbolic query into a **compact, ranked,
+token-budgeted** set of file/line ranges for Claude to read. It is hybrid: multiple independent
+retrievers run, their results are fused and reranked, then trimmed. Graph expansion and MMR are
+available as bounded opt-in signals; the shipped default prioritizes direct evidence.
+
+```
+query
+  │
+  ▼
+[1] intent detection ──────────────┐
+  │                                │ selects retrievers + weights + graph strategy
+  ▼                                ▼
+[2] retrievers (run in parallel)
+     ├─ exact path        (glob/path LIKE)
+     ├─ symbol            (symbols table, exact + fuzzy)
+     ├─ FTS keyword       (FTS5 bm25 over chunks + symbols + summaries)
+     └─ vector  (opt-in)  (sqlite-vec / local embeddings)
+  │
+  ▼
+[3] rank fusion  ── Reciprocal Rank Fusion (RRF) across retriever result lists
+  │
+  ▼
+[4] rerank  ── feature-based score (symbol-kind, path, source role, centrality)
+  │
+  ▼
+[5] optional graph expansion  ── pull in imports/callers/callees per intent (bounded)
+  │
+  ▼
+[6] diversity / duplicate filtering (MMR opt-in, SimHash duplicate guard)
+  │
+  ▼
+[7] token budgeting  ── greedy fill under --token-budget; snippets trimmed + secret-redacted
+  │
+  ▼
+ranked results + recommended_reads + fallback_suggestions
+```
+
+## 1. Intent detection (`retrieval/intent.py`)
+
+Cheap, rule-first classifier (regex/keyword heuristics; optionally a tiny local model later). Each
+intent maps to a retriever mix, a graph strategy, and an output shape.
+
+| Intent | Trigger examples | Retriever emphasis | Graph strategy |
+|---|---|---|---|
+| `locate_impl` | "where is X implemented", "find the X function" | symbol > path > fts | none / defs only |
+| `how_it_works` | "how does X work", "explain X flow" | fts + symbol + vector | expand callees 1–2 hops |
+| `impact` | "what breaks if I change X", "what depends on X" | symbol + path | expand callers/importers (up) |
+| `find_refs` | "find references to X", "who calls X" | symbol | edges where target=X |
+| `data_flow` | "trace data flow of X", "where does X get set" | symbol + fts | follow assignments/calls both ways |
+| `debug_error` | pasted stack trace / "error: ...", "why does X fail" | fts (error string) + symbol | expand around match |
+| `architecture` | "explain the architecture", "high-level overview" | summaries + graph centrality | package/module summaries |
+| `keyword` | fallback when nothing matches above | fts + vector | none |
+
+Each intent also sets a **default token budget** and whether to return module/package summaries
+instead of code (e.g. `architecture` returns summaries first).
+
+## 2. Retrievers (`retrieval/searchers.py`)
+
+All retrievers return a uniform `Candidate(id, kind, path, line_start, line_end, symbol, score,
+source)` list so fusion is source-agnostic.
+
+- **Path** — exact and glob path matches (`src/auth/*.py`, `auth.py`). Highest precision; surfaced
+  first when the query clearly names a path.
+- **Symbol** — query against `symbols` (exact, identifier parts, bounded fuzzy matching). Carries
+  `kind` (function/class/method/...) and signature. Primary for `locate_impl` / `find_refs`.
+  Fuzzy identifier matching (acronym / concatenation / edit distance) runs only as a **recall
+  fallback**, when the precise lookup named no symbol and returned fewer than
+  `fuzzy_fallback_min` rows. Measured over 305 queries on three repositories it moved no ranking
+  metric while costing ~20% of query latency, so it is kept for typos and abbreviations but no
+  longer runs when the query already spelled its identifier correctly.
+- **FTS** — FTS5 `bm25()` over the `fts_chunks` virtual table (chunk text + symbol names +
+  summaries indexed). Query-time camelCase/snake_case splitting, small down-weighted synonym
+  expansion, and soft coverage scoring make natural-language questions robust without weakening
+  exact terms.
+- **Vector** *(opt-in)* — cosine similarity over chunk embeddings via `sqlite-vec`. Only runs if
+  `embeddings.enabled = true`. Adds semantic recall for paraphrased queries. Absent → pipeline
+  degrades gracefully to FTS+symbol.
+
+## 3. Rank fusion (`retrieval/fusion.py`)
+
+**Reciprocal Rank Fusion** combines the per-retriever ranked lists without needing comparable raw
+scores:
+
+```
+RRF(d) = Σ_r  w_r · k / (k + rank_r(d))    # k ≈ 60, w_r = per-intent retriever weight
+```
+
+- Robust to scale differences between BM25 and cosine.
+- Per-intent weights `w_r` let `locate_impl` favor the symbol list and `how_it_works` favor FTS.
+- Scaled by `k` so fused scores and the reranker's bounded bonuses share an O(1) scale. This is a
+  monotonic rescale; fusion order is unchanged.
+- Ties broken by rerank features (next step).
+
+### Cross-locator file agreement
+
+Fusion keys on `(path, line-bucket)`, not `(path, start, end)`, because different retrievers report
+different line ranges for the same place. Bucketing alone was not enough: a symbol defined at line
+40 and a lexical hit at line 120 are genuinely different locators, so a file that **two retrievers
+agreed on** still fused as two separate candidates, each carrying one retriever's evidence — and
+cross-source agreement, the entire point of RRF, never fired.
+
+Each candidate therefore also receives, at weight `file_agreement_weight`, the RRF mass of every
+retriever that found its *file* at some other locator:
+
+```
+score(d) = RRF(d) + α · Σ_{r ∉ sources(d)}  w_r · k / (k + best_rank_r(path(d)))
+```
+
+Retrievers already counted at the candidate's own locator are excluded, so nothing double-counts,
+and the term is bounded by the same weights as fusion itself. `α = 0.4`; the 0.3–0.6 plateau peaks
+there. Set `RetrievalTuning(file_agreement=False)` to recover plain locator-only RRF.
+
+## 4. Reranking (`retrieval/rerank.py`)
+
+A lightweight, explainable feature score (no external model required) layered on the fused order.
+Every term is bounded, so reranking reorders near-neighbours rather than overruling retrieval:
+
+| Feature | Effect | Intuition |
+|---|---:|---|
+| Exact symbol match | +0.20 | the user named a specific symbol |
+| Symbol definition kind | +0.05 | a `def`/`class` outranks an incidental mention |
+| Symbol name among query terms | +0.05 | the name was asked for, not just matched |
+| Path term match | +0.05 | the user supplied a location clue |
+| Graph centrality (`in_degree`) | ≤ +0.08 | `log1p`-damped, so a god class cannot dominate |
+| Reference-count fallback | ≤ +0.04 | for names too common to resolve a precise `in_degree` |
+| Source role prior | −0.25…+0.08 | see below |
+| Generated, or test on a non-test query | −0.15 | supporting evidence, not the answer |
+
+### Source role priors (`retrieval/priors.py`)
+
+| Role | Prior | Rationale |
+|---|---:|---|
+| Implementation | +0.08 | the answer to a code question is usually code |
+| Test | −0.06 | flips to +0.05 when the query or intent is test-oriented |
+| Documentation | −0.20 | prose *about* a feature matches a natural-language question more literally than the code implementing it, so design notes and plans crowded out the modules they describe |
+| Generated / vendor / build | −0.25 | never the answer; kept strictly below documentation |
+
+The documentation prior deepened from −0.05 in 1.8.0. Because it only reorders prose relative to
+code — never below other prose — documentation-seeking queries improved too (MRR 0.579 → 0.612).
+At −0.35 that reverses and the `docs` category collapses, so the optimum is interior, not a
+"more is better" knob. `MAX_ABS_PRIOR` caps every prior so this stays a tiebreaker.
+
+The reranker also produces the human-readable **`reason`** string per result
+(e.g. *"exact symbol match · 4 callers · in src/auth/"*).
+
+## 5. Graph expansion (`graph/retrieval.py`; `graph/expand.py` for impact APIs)
+
+Graph expansion runs only when the tuning enables `graph_source` and the intent plan requests a
+graph strategy. It is disabled by default because the reproducible self-repository ablation
+reduced direct-hit MRR when related nodes displaced lexical hits.
+
+When enabled, it is bounded by depth and node cap:
+
+- `impact` → walk **up** edges (callers, importers) = blast radius.
+- `how_it_works` → walk **down** edges (callees, imported defs) = mechanism.
+- `find_refs` → walk **up** edges to callers/importers.
+- `data_flow` → walk **both** directions along call/assignment edges.
+
+Expanded nodes retain edge confidence and receive distance-decayed scores so seeds stay on top.
+
+## 6. Diversity and duplicate control
+
+`retrieval.diversity` provides bounded MMR selection and SimHash near-duplicate suppression.
+
+MMR is disabled in the shipped default: it moved no ranking metric on the benchmark and roughly
+doubled p50 latency. Callers that need broader snippet coverage can enable
+`RetrievalTuning(mmr=True)`.
+
+SimHash duplicate suppression stays on, but on noise grounds rather than ranking grounds: it does
+not move MRR, and it takes the duplicate rate of returned snippets from ~1.6% to ~0%. The
+over-fetch that feeds selection is now an explicit `candidate_pool_multiplier` rather than an
+implicit side effect of enabling dedup — an earlier ablation credited dedup with a quality win that
+was really the wider pool doing the work.
+
+## 7. Token budgeting (`retrieval/budget.py`)
+
+Results are trimmed to fit `--token-budget` (default per intent, e.g. 1500 tokens):
+
+1. Always include result metadata (path, line range, symbol, reason) — cheap.
+2. Greedily attach snippets to the highest-ranked results until budget is hit.
+3. Snippets are trimmed to the relevant line range (± a few context lines), not whole functions.
+4. Lower-ranked results become **`recommended_reads`** (path + range, no snippet) so Claude can
+   choose to read them itself.
+5. Snippet text passes through secret redaction (see SECURITY.md) before emission.
+
+The point: Claude gets enough to decide, and a precise list of what to read next — never a dump.
+
+## 8. Confidence & fallback
+
+A `confidence` score (high/medium/low) is derived from: top RRF score, score gap between #1 and #2,
+number of agreeing retrievers, and whether a symbol matched exactly.
+
+- **high** → Claude reads `recommended_reads` and answers.
+- **medium** → Claude reads, but may verify with one Grep.
+- **low** → skill instructs Claude to **fall back** to `ripgrep`/Grep/Glob with suggested patterns
+  emitted in `fallback_suggestions` (derived from query terms + detected symbols).
+
+## 9. Output payload (shared by Markdown + JSON)
+
+```jsonc
+{
+  "query": "where is auth token refresh implemented",
+  "intent": "locate_impl",
+  "confidence": "high",
+  "results": [
+    {
+      "rank": 1,
+      "path": "src/auth/token.py",
+      "line_start": 88, "line_end": 134,
+      "symbols": ["refresh_access_token"],
+      "score": 0.91,
+      "reason": "exact symbol match · 4 callers · in src/auth/",
+      "snippet": "def refresh_access_token(...):\n    ..."
+    }
+  ],
+  "recommended_reads": [
+    { "path": "src/auth/token.py", "line_start": 88, "line_end": 134 },
+    { "path": "src/auth/middleware.py", "line_start": 40, "line_end": 72 }
+  ],
+  "fallback_suggestions": {
+    "ripgrep": ["rg -n \"refresh_access_token\" src/", "rg -n \"token.*refresh\""]
+  }
+}
+```
+
+The Markdown renderer (`output/markdown.py`) prints the same data as a tight table + fenced
+snippets so it's compact in Claude's context. See SKILL.md for how Claude is told to read it.