codebase-index 1.8.0__tar.gz → 1.9.0__tar.gz

codebase_index-1.9.0/.claude/skills/codebase-index/.skill_version

@@ -0,0 +1 @@
+1.9.0

{codebase_index-1.8.0 → codebase_index-1.9.0}/.claude-plugin/plugin.json

@@ -3,7 +3,7 @@
   "name": "codebase-index",
   "displayName": "Codebase Index",
   "description": "Give Claude a precise local map of your codebase: find implementations, trace behavior, and predict change impact with file-line evidence.",
-  "version": "1.8.0",
+  "version": "1.9.0",
   "author": {
     "name": "codebase-index contributors"
   },

codebase_index-1.9.0/.codex/skills/codebase-index/.skill_version

@@ -0,0 +1 @@
+1.9.0

codebase_index-1.9.0/.opencode/skills/codebase-index/.skill_version

@@ -0,0 +1 @@
+1.9.0

{codebase_index-1.8.0 → codebase_index-1.9.0}/CHANGELOG.md

@@ -6,6 +6,89 @@ All notable changes to this project are documented here. The format is based on
 
 ## [Unreleased]
 
+## [1.9.0] - 2026-09-02
+
+### Added
+
+- **Objective ground truth from git history.** `tests/eval/gen_queries.py` mints a
+  retrieval benchmark from any git repository by pairing a human-written commit
+  subject with the files that commit actually changed. Unlike docstring-derived
+  benchmarks the query text is not copied into the document being retrieved, so the
+  set is leak-free by construction; merges, reverts, releases, version bumps,
+  sweeping refactors, changelog-style answers and benchmark scaffolding are all
+  filtered out.
+- **Multi-corpus, multi-language evaluation.** `run_eval.py --corpus REPO:QUERIES`
+  pools several repositories into one benchmark so ranking changes are validated
+  outside this repository and outside Python. The 1.9.0 defaults were measured on
+  305 queries across Python, Java and TypeScript corpora.
+- **Significance testing.** Every non-baseline row now reports a paired bootstrap
+  95% confidence interval and a paired permutation p-value (both seeded, therefore
+  reproducible). Query sets of this size have a noise floor of several MRR points;
+  signals now ship on the strength of that test rather than the sign of a delta.
+- **Context-noise metrics.** The report adds mean emitted snippet tokens, duplicate
+  rate of returned results, and p99 latency alongside the existing IR metrics.
+
+### Changed
+
+- **Fusion now scores cross-retriever agreement.** RRF fuses on
+  `(path, line-bucket)`, so a symbol definition at line 40 and a lexical hit at line
+  120 in the same file fused as two unrelated candidates — two retrievers agreeing
+  on a file produced two weak results instead of one strong one, and cross-source
+  agreement never reached the score. Each candidate now also receives, at
+  `file_agreement_weight` (0.4), the RRF mass of every retriever that found its file
+  at another locator, excluding retrievers already counted at that locator.
+  Ablatable via `RetrievalTuning(file_agreement=False)`.
+- **Documentation demotion deepened** from -0.05 to -0.20. Prose describing a
+  feature matches a natural-language question more literally than the code
+  implementing it, so design notes and plans were displacing the modules they
+  describe. Because this only reorders prose relative to code, documentation-seeking
+  queries improved as well (category MRR 0.579 → 0.612). Generated/vendor paths move
+  to -0.25 so they remain the least-preferred role, and `MAX_ABS_PRIOR` now pins the
+  invariant that priors stay tiebreakers.
+- **Fuzzy identifier matching is now a recall fallback.** It runs only when the
+  precise symbol lookup named no symbol and returned fewer than
+  `fuzzy_fallback_min` (3) rows. It moved no ranking metric across 305 queries while
+  accounting for ~20% of query latency; typo and acronym recall is unchanged because
+  those are exactly the queries where the precise lookup comes up empty.
+- **Candidate over-fetch is explicit.** `candidate_pool_multiplier` (default 2)
+  replaces the implicit widening that happened whenever dedup or MMR was enabled.
+  Making it explicit revealed that the quality previously credited to SimHash dedup
+  was really the wider pool; dedup is retained for what it does measurably do, which
+  is cutting the duplicate rate of returned snippets from ~1.6% to ~0%.
+
+### Fixed
+
+- **Synonym matches were reported as exact symbol matches.** `is_exact` came from
+  SQL and was relative to whichever needle retrieved the row, so a synonym
+  expansion ("config" for "configuration") marked an unrelated symbol as an exact
+  match — worth a +0.20 rerank bonus and an unconditional `high` confidence.
+  Exactness is now judged against the terms the user actually typed.
+- **Duplicate suppression was order-dependent on ties.** Equal-scoring duplicates
+  handed the slot to whichever copy arrived last, contradicting the documented
+  "ties favor input order" and making the retained snippet depend on retriever
+  emission order.
+
+### Performance
+
+- Query latency roughly halves: p50 78.6 ms → 51.2 ms, p95 193.2 ms → 94.9 ms,
+  p99 277.1 ms → 152.2 ms on the pooled three-repository benchmark, from the fuzzy
+  fallback plus a SimHash fingerprint that folds repeated tokens by multiplicity and
+  caches token digests. Fingerprint output is bit-for-bit unchanged.
+- Mean emitted snippet tokens fall 1183 → 1091 per query.
+
+### Retrieval quality
+
+Pooled over 305 queries (Python, Java, TypeScript), v1.8.0 → 1.9.0:
+
+| Metric | v1.8.0 | 1.9.0 | Δ | p |
+|---|---|---|---|---|
+| MRR | 0.564 | 0.591 | +0.027 | <0.001 |
+| MAP | 0.433 | 0.461 | +0.028 | <0.001 |
+| nDCG@10 | 0.503 | 0.527 | +0.024 | <0.001 |
+| recall@5 | 0.529 | 0.560 | +0.031 | <0.001 |
+| P@5 | 0.182 | 0.192 | +0.009 | 0.001 |
+| hit@3 | 0.649 | 0.666 | +0.016 | 0.124 |
+
 ## [1.8.0] - 2026-09-02
 
 ### Added

{codebase_index-1.8.0 → codebase_index-1.9.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: codebase-index
-Version: 1.8.0
+Version: 1.9.0
 Summary: A precise local codebase map for AI coding agents: find, trace, and predict with file-line evidence.
 Project-URL: Homepage, https://github.com/denfry/codebase-index
 Project-URL: Documentation, https://github.com/denfry/codebase-index/tree/main/docs
@@ -321,7 +321,7 @@ See [MCP.md](docs/MCP.md) for client configuration.
 
 ## Project status
 
-The latest released line is **1.8.0**. It includes:
+The latest released line is **1.9.0**. It includes:
 
 - hybrid and optional vector retrieval;
 - Tree-sitter symbol extraction across the documented language tiers;
@@ -330,15 +330,21 @@ The latest released line is **1.8.0**. It includes:
 - shortest dependency paths and node descriptions;
 - token-budgeted and skeletonized retrieval packets;
 - benchmark-calibrated lexical expansion, fuzzy identifier matching, and source-aware ranking;
+- rank fusion that scores cross-retriever agreement at file level, not just at a locator;
 - bounded, intent-directed graph discovery with optional diversity and duplicate suppression;
 - CLI, Skill, plugin, and MCP delivery;
 - incremental updates, watch hooks, diagnostics, skill rollback, and diff-aware
-  impact analysis.
-
-Planned work is deliberately separated from shipped capability. The next
-product priorities are stronger real-repository evaluations, typed framework
-edges, and an even more direct task-context workflow. See the
-[roadmap](docs/ROADMAP.md).
+  impact analysis;
+- a multi-repository retrieval evaluation with leak-free git-derived ground truth,
+  one-signal ablations, and paired significance tests
+  ([tests/eval](tests/eval/README.md)).
+
+Every shipped ranking signal has to survive that evaluation: 1.9.0 removed the
+cost of two signals that could not demonstrate a benefit and rejected several
+plausible ones outright (IDF-weighted coverage, stemming, graph propagation, MMR,
+a file-length prior). Planned work is deliberately separated from shipped
+capability. The next product priorities are typed framework edges and an even more
+direct task-context workflow. See the [roadmap](docs/ROADMAP.md).
 
 ## Documentation
 

{codebase_index-1.8.0 → codebase_index-1.9.0}/README.md

@@ -271,7 +271,7 @@ See [MCP.md](docs/MCP.md) for client configuration.
 
 ## Project status
 
-The latest released line is **1.8.0**. It includes:
+The latest released line is **1.9.0**. It includes:
 
 - hybrid and optional vector retrieval;
 - Tree-sitter symbol extraction across the documented language tiers;
@@ -280,15 +280,21 @@ The latest released line is **1.8.0**. It includes:
 - shortest dependency paths and node descriptions;
 - token-budgeted and skeletonized retrieval packets;
 - benchmark-calibrated lexical expansion, fuzzy identifier matching, and source-aware ranking;
+- rank fusion that scores cross-retriever agreement at file level, not just at a locator;
 - bounded, intent-directed graph discovery with optional diversity and duplicate suppression;
 - CLI, Skill, plugin, and MCP delivery;
 - incremental updates, watch hooks, diagnostics, skill rollback, and diff-aware
-  impact analysis.
-
-Planned work is deliberately separated from shipped capability. The next
-product priorities are stronger real-repository evaluations, typed framework
-edges, and an even more direct task-context workflow. See the
-[roadmap](docs/ROADMAP.md).
+  impact analysis;
+- a multi-repository retrieval evaluation with leak-free git-derived ground truth,
+  one-signal ablations, and paired significance tests
+  ([tests/eval](tests/eval/README.md)).
+
+Every shipped ranking signal has to survive that evaluation: 1.9.0 removed the
+cost of two signals that could not demonstrate a benefit and rejected several
+plausible ones outright (IDF-weighted coverage, stemming, graph propagation, MMR,
+a file-length prior). Planned work is deliberately separated from shipped
+capability. The next product priorities are typed framework edges and an even more
+direct task-context workflow. See the [roadmap](docs/ROADMAP.md).
 
 ## Documentation
 

{codebase_index-1.8.0 → codebase_index-1.9.0}/docs/ARCHITECTURE.md

@@ -2,7 +2,7 @@
 
 ## 1. Overview
 
-`codebase-index` is a **local-first** code intelligence layer for AI coding agents. In `1.8.0`
+`codebase-index` is a **local-first** code intelligence layer for AI coding agents. In `1.9.0`
 it has two shipped faces:
 
 1. **A Claude Code Skill** (`.claude/skills/codebase-index/SKILL.md`) that Claude auto-invokes for
@@ -97,8 +97,9 @@ codebase-index/
     ├── graph/                   # builder.py (edge resolution), expand.py (impact),
     │                            #   export.py (HTML graph)
     ├── storage/                 # db.py (pragmas, schema, version guard), schema.sql, repo.py
-    ├── retrieval/               # intent.py, searchers.py, fusion.py, rerank.py,
-    │                            #   budget.py, pipeline.py, types.py
+    ├── retrieval/               # intent.py, searchers.py, fusion.py, rerank.py, priors.py,
+    │                            #   lexical.py, fuzzy.py, diversity.py, skeleton.py,
+    │                            #   tuning.py (ablation contract), budget.py, pipeline.py, types.py
     ├── embeddings/              # backend.py, noop.py (default), local.py, external.py — opt-in
     ├── output/                  # markdown.py, json.py, redact.py
     ├── watch/                   # watcher.py (optional, watchdog-based)

{codebase_index-1.8.0 → codebase_index-1.9.0}/docs/BENCHMARKS.md

@@ -1,10 +1,11 @@
 # Benchmarks
 
-`codebase-index` has three benchmark surfaces. Read them with their status in
+`codebase-index` has four benchmark surfaces. Read them with their status in
 mind — the whole point of this page is to keep evidence and aspiration separate.
 
 | Surface | What it is | Status | Use it as |
 |---|---|---|---|
+| Retrieval eval (`tests/eval/`) | Ranking quality vs a fixed baseline across multiple real repositories, with significance tests | **Proven (relative)** | The gate for ranking changes; measures *deltas*, not absolute superiority |
 | Public suite (`tests/benchmark_public.py`) | Deterministic synthetic multi-language fixture with the full metric framework | **Toy/synthetic** | CI regression gate + metric shape, **not** product-quality evidence |
 | Smoke/perf (`test_perf_smoke.py`, `test_benchmark_comparison.py`) | Latency + output-size guards on a tiny fixture | **Toy/smoke** | Regression checks only |
 | Honest real-repo (`tests/benchmark_honest.py`) | 55k LOC Java repo, recall@3 vs disciplined `rg` baseline, symmetric token accounting | **Proven (one repo)** | The only headline product-quality number we stand behind today |
@@ -15,15 +16,33 @@ Do not write, imply, or ship any of these until a run with published logs exists
 
 - Any 10k / 100k / 1M LOC scale or speed claim (no real run at that size).
 - "Beats Cursor / Sourcegraph / Codebase-Memory MCP" — no head-to-head exists.
-- Per-language quality claims beyond Java (the honest run is Java-only).
+- Per-language *absolute* quality claims beyond Java. The retrieval eval covers
+  Python, Java and TypeScript, but it measures this system against its own earlier
+  versions — it says a change helped, not that the product beats an alternative.
 - Generic "Nx faster" / "Nx fewer tokens" without naming the baseline and repo.
-- Latency claims — the honest run explicitly does not headline latency
-  (Python process start dominates; real `rg` is tens of ms).
+- Latency claims against external tools — the honest run explicitly does not
+  headline latency (Python process start dominates; real `rg` is tens of ms).
+  Version-over-version latency from the retrieval eval is in-process and is only
+  comparable to other runs of that harness.
 
 The defensible headline today is exactly: **on one 55k LOC Java repo, recall@3 was
 70% (index) vs 40% (`rg`+window), using ~13× fewer answer tokens.** Everything
 else is roadmap.
 
+## Retrieval evaluation (ranking gate)
+
+See [tests/eval/README.md](../tests/eval/README.md) for the full protocol. In short:
+ground truth comes from hand-written queries verified against the source tree and
+from commit-subject → changed-files pairs mined from git history (leak-free: the
+query text is not in the indexed corpus). Corpora are pooled across languages, one
+index per corpus is shared by every variant, and every non-baseline row carries a
+paired bootstrap CI and permutation p-value.
+
+`1.9.0` was measured over 305 queries across Python, Java and TypeScript corpora
+against `1.8.0`: MRR +0.027, MAP +0.028, nDCG@10 +0.024, recall@5 +0.031 (all
+p < 0.001), with p50 latency 78.6 ms → 51.2 ms. These are version-over-version
+ranking deltas on those corpora, not a universal quality claim.
+
 ## Public benchmark suite
 
 Run:

{codebase_index-1.8.0 → codebase_index-1.9.0}/docs/RETRIEVAL.md

@@ -65,6 +65,11 @@ source)` list so fusion is source-agnostic.
   first when the query clearly names a path.
 - **Symbol** — query against `symbols` (exact, identifier parts, bounded fuzzy matching). Carries
   `kind` (function/class/method/...) and signature. Primary for `locate_impl` / `find_refs`.
+  Fuzzy identifier matching (acronym / concatenation / edit distance) runs only as a **recall
+  fallback**, when the precise lookup named no symbol and returned fewer than
+  `fuzzy_fallback_min` rows. Measured over 305 queries on three repositories it moved no ranking
+  metric while costing ~20% of query latency, so it is kept for typos and abbreviations but no
+  longer runs when the query already spelled its identifier correctly.
 - **FTS** — FTS5 `bm25()` over the `fts_chunks` virtual table (chunk text + symbol names +
   summaries indexed). Query-time camelCase/snake_case splitting, small down-weighted synonym
   expansion, and soft coverage scoring make natural-language questions robust without weakening
@@ -79,28 +84,66 @@ source)` list so fusion is source-agnostic.
 scores:
 
 ```
-RRF(d) = Σ_r  w_r / (k + rank_r(d))        # k ≈ 60, w_r = per-intent retriever weight
+RRF(d) = Σ_r  w_r · k / (k + rank_r(d))    # k ≈ 60, w_r = per-intent retriever weight
 ```
 
 - Robust to scale differences between BM25 and cosine.
 - Per-intent weights `w_r` let `locate_impl` favor the symbol list and `how_it_works` favor FTS.
+- Scaled by `k` so fused scores and the reranker's bounded bonuses share an O(1) scale. This is a
+  monotonic rescale; fusion order is unchanged.
 - Ties broken by rerank features (next step).
 
-## 4. Reranking (`retrieval/rerank.py`)
+### Cross-locator file agreement
+
+Fusion keys on `(path, line-bucket)`, not `(path, start, end)`, because different retrievers report
+different line ranges for the same place. Bucketing alone was not enough: a symbol defined at line
+40 and a lexical hit at line 120 are genuinely different locators, so a file that **two retrievers
+agreed on** still fused as two separate candidates, each carrying one retriever's evidence — and
+cross-source agreement, the entire point of RRF, never fired.
+
+Each candidate therefore also receives, at weight `file_agreement_weight`, the RRF mass of every
+retriever that found its *file* at some other locator:
+
+```
+score(d) = RRF(d) + α · Σ_{r ∉ sources(d)}  w_r · k / (k + best_rank_r(path(d)))
+```
+
+Retrievers already counted at the candidate's own locator are excluded, so nothing double-counts,
+and the term is bounded by the same weights as fusion itself. `α = 0.4`; the 0.3–0.6 plateau peaks
+there. Set `RetrievalTuning(file_agreement=False)` to recover plain locator-only RRF.
 
-A lightweight, explainable feature score (no external model required) layered on the fused order:
+## 4. Reranking (`retrieval/rerank.py`)
 
-| Feature | Intuition |
-|---|---|
-| symbol-kind match | a `def`/`class` outranks an incidental mention |
-| path proximity | files near a query-named path score higher |
-| graph centrality | high in/out-degree nodes matter more for `architecture` |
-| recency | recently changed files (git mtime) slightly boosted |
-| exact-name bonus | exact symbol-name match dominates fuzzy |
-| test/generated penalty | test files and generated code demoted unless asked |
+A lightweight, explainable feature score (no external model required) layered on the fused order.
+Every term is bounded, so reranking reorders near-neighbours rather than overruling retrieval:
+
+| Feature | Effect | Intuition |
+|---|---:|---|
+| Exact symbol match | +0.20 | the user named a specific symbol |
+| Symbol definition kind | +0.05 | a `def`/`class` outranks an incidental mention |
+| Symbol name among query terms | +0.05 | the name was asked for, not just matched |
+| Path term match | +0.05 | the user supplied a location clue |
+| Graph centrality (`in_degree`) | ≤ +0.08 | `log1p`-damped, so a god class cannot dominate |
+| Reference-count fallback | ≤ +0.04 | for names too common to resolve a precise `in_degree` |
+| Source role prior | −0.25…+0.08 | see below |
+| Generated, or test on a non-test query | −0.15 | supporting evidence, not the answer |
+
+### Source role priors (`retrieval/priors.py`)
+
+| Role | Prior | Rationale |
+|---|---:|---|
+| Implementation | +0.08 | the answer to a code question is usually code |
+| Test | −0.06 | flips to +0.05 when the query or intent is test-oriented |
+| Documentation | −0.20 | prose *about* a feature matches a natural-language question more literally than the code implementing it, so design notes and plans crowded out the modules they describe |
+| Generated / vendor / build | −0.25 | never the answer; kept strictly below documentation |
+
+The documentation prior deepened from −0.05 in 1.8.0. Because it only reorders prose relative to
+code — never below other prose — documentation-seeking queries improved too (MRR 0.579 → 0.612).
+At −0.35 that reverses and the `docs` category collapses, so the optimum is interior, not a
+"more is better" knob. `MAX_ABS_PRIOR` caps every prior so this stays a tiebreaker.
 
 The reranker also produces the human-readable **`reason`** string per result
-(e.g. *"exact symbol match · called by 4 sites · in src/auth/"*).
+(e.g. *"exact symbol match · 4 callers · in src/auth/"*).
 
 ## 5. Graph expansion (`graph/retrieval.py`; `graph/expand.py` for impact APIs)
 
@@ -119,11 +162,17 @@ Expanded nodes retain edge confidence and receive distance-decayed scores so see
 
 ## 6. Diversity and duplicate control
 
-`retrieval.diversity` provides bounded MMR selection and SimHash near-duplicate
-suppression. MMR is disabled in the shipped default because the reproducible
-benchmark favored relevance-only ranking; callers that need broader snippet
-coverage can enable `RetrievalTuning(mmr=True)`. Duplicate suppression remains
-available independently.
+`retrieval.diversity` provides bounded MMR selection and SimHash near-duplicate suppression.
+
+MMR is disabled in the shipped default: it moved no ranking metric on the benchmark and roughly
+doubled p50 latency. Callers that need broader snippet coverage can enable
+`RetrievalTuning(mmr=True)`.
+
+SimHash duplicate suppression stays on, but on noise grounds rather than ranking grounds: it does
+not move MRR, and it takes the duplicate rate of returned snippets from ~1.6% to ~0%. The
+over-fetch that feeds selection is now an explicit `candidate_pool_multiplier` rather than an
+implicit side effect of enabling dedup — an earlier ablation credited dedup with a quality win that
+was really the wider pool doing the work.
 
 ## 7. Token budgeting (`retrieval/budget.py`)
 

{codebase_index-1.8.0 → codebase_index-1.9.0}/docs/RETRIEVAL_PIPELINE.md

@@ -97,17 +97,21 @@ direct lexical or symbol evidence.
 
 ## 6. Diversity and duplicate control
 
-SimHash suppresses near-duplicate snippets independently of MMR. Bounded
-Maximal Marginal Relevance is available through `RetrievalTuning(mmr=True)`;
-the shipped default keeps relevance-only ordering because the benchmark favored
-direct hits.
+SimHash suppresses near-duplicate snippets independently of MMR. It does not move
+ranking metrics; it earns its place by taking the duplicate rate of returned
+snippets from ~1.6% to ~0%. Bounded Maximal Marginal Relevance is available
+through `RetrievalTuning(mmr=True)`; the shipped default keeps relevance-only
+ordering because MMR moved no metric and roughly doubled p50 latency.
+
+The over-fetch that feeds these selection stages is `candidate_pool_multiplier`
+(default 2), an explicit knob rather than a side effect of enabling dedup.
 
 ## 7. Reciprocal Rank Fusion (RRF)
 
 Combines ranked lists from the enabled retrievers:
 
 ```
-RRF_score(d) = Σ w_r · k / (k + rank_r(d))
+RRF_score(d) = Σ_r w_r · k / (k + rank_r(d))
 ```
 
 The implementation multiplies textbook RRF by `k` so fusion and bounded rerank
@@ -118,8 +122,13 @@ Where:
 - `rank_r(d)` is the rank of document `d` in retriever `r`
 - `w_r` is the intent/tuning weight for retriever `r`
 
-The implementation merges co-located chunks into one per-file bucket before
-fusion, preventing a large file from dominating the result list.
+Candidates are keyed by `(path, line-bucket)`, so co-located hits merge. Because a
+symbol definition and a lexical hit in the same file are often *not* co-located,
+each candidate additionally receives — at `file_agreement_weight` (0.4) — the RRF
+mass of every retriever that found its file at a different locator. Retrievers
+already counted at that locator are excluded, so nothing double-counts. Without
+this, two retrievers agreeing on a file produced two weak candidates instead of one
+strong one, and cross-source agreement never affected the score.
 
 ## 8. Reranking
 
@@ -132,8 +141,8 @@ After fusion, apply bounded explainable boosts and penalties:
 | Path term match | +0.05 | User supplied a location clue |
 | Degree / reference evidence | up to +0.08 | Stable structural tiebreaker |
 | Implementation source prior | +0.08 | Prefer source over prose/tests |
-| Documentation source prior | -0.05 | Avoid docs displacing implementation |
-| Generated/vendor/build | -0.12 | Suppress low-value derived code |
+| Documentation source prior | -0.20 | Prose describing a feature outmatches the code lexically |
+| Generated/vendor/build | -0.25 | Suppress low-value derived code |
 | Test path on non-test query | -0.06 | Keep tests as supporting evidence |
 
 ## 9. Confidence

{codebase_index-1.8.0 → codebase_index-1.9.0}/requirements.lock

@@ -1,3 +1,3 @@
-codebase-index @ https://github.com/denfry/codebase-index/archive/refs/tags/v1.8.0.tar.gz
+codebase-index @ https://github.com/denfry/codebase-index/archive/refs/tags/v1.9.0.tar.gz
 tree-sitter==0.25.2
 tree-sitter-language-pack==1.8.1

{codebase_index-1.8.0 → codebase_index-1.9.0}/src/codebase_index/__init__.py

@@ -4,4 +4,4 @@ Public surface is intentionally small; the CLI (`codebase_index.cli:app`) is the
 See docs/ARCHITECTURE.md for the module map.
 """
 
-__version__ = "1.8.0"
+__version__ = "1.9.0"

{codebase_index-1.8.0 → codebase_index-1.9.0}/src/codebase_index/mcp/server.py

@@ -30,7 +30,7 @@ if TYPE_CHECKING:
     from ..config import Config
 
 try:
-    from mcp.server.fastmcp import FastMCP
+    from mcp.server.fastmcp import FastMCP  # type: ignore[attr-defined]
 except ImportError as exc:  # pragma: no cover
     raise ImportError(
         "MCP server needs the optional extra: pip install codebase-index[mcp]"

{codebase_index-1.8.0 → codebase_index-1.9.0}/src/codebase_index/retrieval/diversity.py

@@ -10,7 +10,9 @@ from __future__ import annotations
 
 import hashlib
 import math
+from collections import Counter
 from collections.abc import Callable, Iterable, Sequence
+from functools import lru_cache
 
 from .types import Candidate
 
@@ -90,6 +92,18 @@ def normalize_code_tokens(content: str | None) -> tuple[str, ...]:
     return tuple(tokens)
 
 
+@lru_cache(maxsize=1 << 16)
+def _token_digest(token: str) -> int:
+    """64-bit blake2b digest of one token. Pure, so caching is safe and bounded.
+
+    Source chunks repeat identifiers and keywords heavily both inside a chunk and
+    across the candidate pool, and hashing dominated fingerprint cost.
+    """
+    return int.from_bytes(
+        hashlib.blake2b(token.encode("utf-8"), digest_size=8).digest(), "big", signed=False
+    )
+
+
 def token_fingerprint(tokens: Iterable[str] | str | None) -> int:
     """Compute a stable unsigned 64-bit SimHash for tokens or source content."""
     if tokens is None:
@@ -100,12 +114,14 @@ def token_fingerprint(tokens: Iterable[str] | str | None) -> int:
     if not values:
         return 0
 
+    # Weighting distinct tokens by multiplicity is arithmetically identical to
+    # summing each occurrence, and collapses the per-occurrence hash and 64-bit
+    # accumulation loop to per-distinct-token work. Output is bit-for-bit unchanged.
     weights = [0] * 64
-    for token in values:
-        digest = hashlib.blake2b(str(token).encode("utf-8"), digest_size=8).digest()
-        hashed = int.from_bytes(digest, "big", signed=False)
+    for token, count in Counter(str(value) for value in values).items():
+        hashed = _token_digest(token)
         for bit in range(64):
-            weights[bit] += 1 if (hashed >> bit) & 1 else -1
+            weights[bit] += count if (hashed >> bit) & 1 else -count
 
     fingerprint = 0
     for bit, weight in enumerate(weights):
@@ -133,8 +149,14 @@ def _score_value(candidate: Candidate) -> float:
 
 
 def _same_or_better(left: Candidate, right: Candidate) -> bool:
-    """Whether left is the representative to retain; ties favor input order."""
-    return _score_value(left) > _score_value(right)
+    """Whether `left` (the incumbent) should stay the representative.
+
+    Ties favor the incumbent, i.e. input order. Candidates arrive sorted by score,
+    so a later duplicate is never strictly better; deciding ties by ">" instead
+    handed the slot to whichever equal-scoring copy happened to arrive last, making
+    the retained snippet depend on retriever emission order.
+    """
+    return _score_value(left) >= _score_value(right)
 
 
 def deduplicate(candidates: Sequence[Candidate], hamming_distance: int = 3) -> list[Candidate]:
@@ -146,14 +168,21 @@ def deduplicate(candidates: Sequence[Candidate], hamming_distance: int = 3) -> l
     fingerprints: list[int | None] = []
 
     for candidate in candidates:
-        _, _, fingerprint = _features(candidate)
+        # Only the fingerprint is needed here; building the token set as well
+        # doubled the per-candidate cost of the pipeline's hottest stage. "No
+        # fingerprint" means "no tokens", never "fingerprint happened to be 0".
+        tokens = normalize_code_tokens(candidate.content)
+        fingerprint = token_fingerprint(tokens) if tokens else None
         if fingerprint is None:
             representatives.append(candidate)
             fingerprints.append(None)
             continue
 
+        # Carry each match's fingerprint alongside its index: it is known non-None
+        # by construction here, and re-reading it from the list later would lose
+        # that guarantee.
         matches = [
-            index
+            (index, existing)
             for index, existing in enumerate(fingerprints)
             if existing is not None and simhash_distance(fingerprint, existing) <= threshold
         ]
@@ -162,17 +191,23 @@ def deduplicate(candidates: Sequence[Candidate], hamming_distance: int = 3) -> l
             fingerprints.append(fingerprint)
             continue
 
-        first = matches[0]
+        indices = [index for index, _ in matches]
+        first = indices[0]
         winner = candidate
-        for index in matches:
+        # The winner is either this candidate or an incumbent, and both
+        # fingerprints are already known — recomputing one from content was pure
+        # duplicated work.
+        winner_fingerprint = fingerprint
+        for index, existing in matches:
             incumbent = representatives[index]
             if _same_or_better(incumbent, winner):
                 winner = incumbent
-        for index in reversed(matches):
+                winner_fingerprint = existing
+        for index in reversed(indices):
             representatives.pop(index)
             fingerprints.pop(index)
         representatives.insert(first, winner)
-        fingerprints.insert(first, token_fingerprint(normalize_code_tokens(winner.content)))
+        fingerprints.insert(first, winner_fingerprint)
 
     return representatives
 

{codebase_index-1.8.0 → codebase_index-1.9.0}/src/codebase_index/retrieval/fusion.py

@@ -13,6 +13,14 @@ Two deliberate departures from the textbook formula:
   retrievers report different line ranges for the same place; an exact key almost
   never coincides across sources, so cross-source agreement — RRF's whole point —
   would never fire. `agreeing_sources` is therefore counted at file granularity.
+* Cross-locator agreement is scored, not merely counted. Bucketing alone does not
+  rescue agreement: a symbol defined at line 40 and a lexical hit at line 120 sit
+  in different buckets of the same file, so two retrievers pointing at one file
+  still fused as two unrelated candidates, each carrying a single retriever's
+  evidence. `file_agreement` adds the missing evidence back at a discount — a
+  candidate also receives, weighted by `file_agreement_weight`, the RRF mass of
+  every retriever that found its *file* somewhere else. Retrievers already
+  counted at the candidate's own locator are excluded, so nothing double-counts.
 
 On merge, the candidate carrying the most signal (symbol > fts > path) is kept as
 the representative so downstream rerank/snippet logic has the richest fields.
@@ -36,11 +44,21 @@ def fuse(
     *,
     weights: dict[str, float],
     k: int,
+    file_agreement: float = 0.0,
 ) -> list[Candidate]:
+    """Fuse per-source ranked lists into one ordered candidate list.
+
+    `file_agreement` is the discount applied to same-file, different-locator
+    evidence; 0.0 reproduces plain locator-only RRF.
+    """
     accum: dict[tuple, float] = {}
     rep: dict[tuple, Candidate] = {}
     seen: set[tuple] = set()
     file_sources: dict[str, set[str]] = {}
+    # Best (lowest) rank each source achieved for each file, and which sources
+    # already contributed at each fused locator.
+    best_file_rank: dict[tuple[str, str], int] = {}
+    key_sources: dict[tuple, set[str]] = {}
 
     for source, candidates in lists.items():
         w = weights.get(source, 0.0)
@@ -48,7 +66,12 @@ def fuse(
             continue
         for rank, cand in enumerate(candidates):
             file_sources.setdefault(cand.path, set()).add(source)
+            file_key = (source, cand.path)
+            previous = best_file_rank.get(file_key)
+            if previous is None or rank < previous:
+                best_file_rank[file_key] = rank
             key = cand.fuse_key()
+            key_sources.setdefault(key, set()).add(source)
             # One contribution per source per locator: a file matching three FTS
             # chunks in the same bucket is one lexical signal, not three.
             if (source, key) in seen:
@@ -57,6 +80,21 @@ def fuse(
             accum[key] = accum.get(key, 0.0) + w * k / (k + rank)
             rep[key] = _richer(rep[key], cand) if key in rep else cand
 
+    if file_agreement > 0.0:
+        for key, base in accum.items():
+            path = rep[key].path
+            own = key_sources[key]
+            extra = 0.0
+            for source in file_sources.get(path, ()):
+                if source in own:
+                    continue
+                w = weights.get(source, 0.0)
+                if w <= 0.0:
+                    continue
+                extra += w * k / (k + best_file_rank[(source, path)])
+            if extra:
+                accum[key] = base + file_agreement * extra
+
     fused = [_replace(rep[key], score=score) for key, score in accum.items()]
     fused.sort(key=lambda c: c.score, reverse=True)
     return [_replace(c, agreeing_sources=len(file_sources[c.path])) for c in fused]