rag-your-code 0.5.0__tar.gz → 0.6.0__tar.gz

{rag_your_code-0.5.0/src/rag_your_code.egg-info → rag_your_code-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rag-your-code
-Version: 0.5.0
+Version: 0.6.0
 Summary: A local, explainable RAG index for codebases and coding agents
 Author: rag-your-code contributors
 License-Expression: MIT
@@ -198,19 +198,49 @@ Chinese, each listing every unit that genuinely answers it
 
 | | generated descriptions | agent-written |
 |---|---|---|
-| hit@1 | 0.171 | **0.500** |
-| hit@3 | 0.314 | **0.729** |
-| MRR | 0.240 | **0.605** |
-| answered with no shared word at all | 15.7% | **0%** |
+| hit@1 | 0.271 | **0.500** |
+| hit@3 | 0.486 | **0.800** |
+| MRR | 0.367 | **0.631** |
+| answered with no shared word at all | 12.9% | **0%** |
 
-Roughly a threefold improvement in first-place accuracy. Nineteen questions
-still fail, which is what makes the set usable for measuring the next change;
-`tests/test_repo_queries.py` asserts that some question always does.
+Roughly double the first-place accuracy. Fourteen questions still fail, which
+is what makes the set usable for measuring the next change;
+`tests/test_repo_queries.py` asserts that some question always does, and that
+the written column beats the generated one.
 
 One failure is worth naming: a query saying `catastrophic backtracking` does
 not reach a description saying `backtracks catastrophically`. There is no
 stemming — exactly the limit documented above.
 
+### Measured on a repository nobody here wrote
+
+The table above is the warmest case this project supports: its own code, its
+own descriptions, and questions written by the same party. It cannot say what
+a first-time user gets. So there is a second ruler — thirty-five questions
+about [cc-enforcer](https://github.com/skymanbp/cc-enforcer), 1153 units, no
+descriptions at all, each question phrased in a user's words rather than in
+the words of the docstring that answers it
+([`benchmarks/cold_queries.json`](benchmarks/cold_queries.json)):
+
+| | before 0.6.0 | now |
+|---|---|---|
+| hit@1 | 0.086 | **0.257** |
+| hit@3 | 0.229 | **0.400** |
+| MRR | 0.157 | **0.314** |
+
+Three times the first-place accuracy, and the same change moved both other
+rulers in the same direction. What it fixed was ranking: scoring used to be
+the fraction of query words a unit contained, so `the` counted for as much as
+`daemon`, and nothing corrected for size — the single largest declaration in
+that repository came back in the top three for four questions out of six. It
+is now BM25 over weighted fields, where a word's worth comes from how rare it
+is in *your* corpus and a word in a declaration's name outweighs the same word
+buried in a body.
+
+Twenty-one of the thirty-five still fail, and the largest remaining cause is
+named in [docs/TESTING.md](docs/TESTING.md): a test declaration often outranks
+the code it tests, because it repeats that code's vocabulary and adds its own.
+
 **What this is:** it moves the semantic work from query time to index time.
 Matching stays lexical. It is LLM-authored keyword expansion, and its reach is
 bounded by how many ways of saying the thing the agent thought to write down.

{rag_your_code-0.5.0 → rag_your_code-0.6.0}/README.md

@@ -171,19 +171,49 @@ Chinese, each listing every unit that genuinely answers it
 
 | | generated descriptions | agent-written |
 |---|---|---|
-| hit@1 | 0.171 | **0.500** |
-| hit@3 | 0.314 | **0.729** |
-| MRR | 0.240 | **0.605** |
-| answered with no shared word at all | 15.7% | **0%** |
+| hit@1 | 0.271 | **0.500** |
+| hit@3 | 0.486 | **0.800** |
+| MRR | 0.367 | **0.631** |
+| answered with no shared word at all | 12.9% | **0%** |
 
-Roughly a threefold improvement in first-place accuracy. Nineteen questions
-still fail, which is what makes the set usable for measuring the next change;
-`tests/test_repo_queries.py` asserts that some question always does.
+Roughly double the first-place accuracy. Fourteen questions still fail, which
+is what makes the set usable for measuring the next change;
+`tests/test_repo_queries.py` asserts that some question always does, and that
+the written column beats the generated one.
 
 One failure is worth naming: a query saying `catastrophic backtracking` does
 not reach a description saying `backtracks catastrophically`. There is no
 stemming — exactly the limit documented above.
 
+### Measured on a repository nobody here wrote
+
+The table above is the warmest case this project supports: its own code, its
+own descriptions, and questions written by the same party. It cannot say what
+a first-time user gets. So there is a second ruler — thirty-five questions
+about [cc-enforcer](https://github.com/skymanbp/cc-enforcer), 1153 units, no
+descriptions at all, each question phrased in a user's words rather than in
+the words of the docstring that answers it
+([`benchmarks/cold_queries.json`](benchmarks/cold_queries.json)):
+
+| | before 0.6.0 | now |
+|---|---|---|
+| hit@1 | 0.086 | **0.257** |
+| hit@3 | 0.229 | **0.400** |
+| MRR | 0.157 | **0.314** |
+
+Three times the first-place accuracy, and the same change moved both other
+rulers in the same direction. What it fixed was ranking: scoring used to be
+the fraction of query words a unit contained, so `the` counted for as much as
+`daemon`, and nothing corrected for size — the single largest declaration in
+that repository came back in the top three for four questions out of six. It
+is now BM25 over weighted fields, where a word's worth comes from how rare it
+is in *your* corpus and a word in a declaration's name outweighs the same word
+buried in a body.
+
+Twenty-one of the thirty-five still fail, and the largest remaining cause is
+named in [docs/TESTING.md](docs/TESTING.md): a test declaration often outranks
+the code it tests, because it repeats that code's vocabulary and adds its own.
+
 **What this is:** it moves the semantic work from query time to index time.
 Matching stays lexical. It is LLM-authored keyword expansion, and its reach is
 bounded by how many ways of saying the thing the agent thought to write down.

{rag_your_code-0.5.0 → rag_your_code-0.6.0}/pyproject.toml

@@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "rag-your-code"
-version = "0.5.0"
+version = "0.6.0"
 description = "A local, explainable RAG index for codebases and coding agents"
 readme = "README.md"
 requires-python = ">=3.10"

{rag_your_code-0.5.0 → rag_your_code-0.6.0/src/rag_your_code.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rag-your-code
-Version: 0.5.0
+Version: 0.6.0
 Summary: A local, explainable RAG index for codebases and coding agents
 Author: rag-your-code contributors
 License-Expression: MIT
@@ -198,19 +198,49 @@ Chinese, each listing every unit that genuinely answers it
 
 | | generated descriptions | agent-written |
 |---|---|---|
-| hit@1 | 0.171 | **0.500** |
-| hit@3 | 0.314 | **0.729** |
-| MRR | 0.240 | **0.605** |
-| answered with no shared word at all | 15.7% | **0%** |
+| hit@1 | 0.271 | **0.500** |
+| hit@3 | 0.486 | **0.800** |
+| MRR | 0.367 | **0.631** |
+| answered with no shared word at all | 12.9% | **0%** |
 
-Roughly a threefold improvement in first-place accuracy. Nineteen questions
-still fail, which is what makes the set usable for measuring the next change;
-`tests/test_repo_queries.py` asserts that some question always does.
+Roughly double the first-place accuracy. Fourteen questions still fail, which
+is what makes the set usable for measuring the next change;
+`tests/test_repo_queries.py` asserts that some question always does, and that
+the written column beats the generated one.
 
 One failure is worth naming: a query saying `catastrophic backtracking` does
 not reach a description saying `backtracks catastrophically`. There is no
 stemming — exactly the limit documented above.
 
+### Measured on a repository nobody here wrote
+
+The table above is the warmest case this project supports: its own code, its
+own descriptions, and questions written by the same party. It cannot say what
+a first-time user gets. So there is a second ruler — thirty-five questions
+about [cc-enforcer](https://github.com/skymanbp/cc-enforcer), 1153 units, no
+descriptions at all, each question phrased in a user's words rather than in
+the words of the docstring that answers it
+([`benchmarks/cold_queries.json`](benchmarks/cold_queries.json)):
+
+| | before 0.6.0 | now |
+|---|---|---|
+| hit@1 | 0.086 | **0.257** |
+| hit@3 | 0.229 | **0.400** |
+| MRR | 0.157 | **0.314** |
+
+Three times the first-place accuracy, and the same change moved both other
+rulers in the same direction. What it fixed was ranking: scoring used to be
+the fraction of query words a unit contained, so `the` counted for as much as
+`daemon`, and nothing corrected for size — the single largest declaration in
+that repository came back in the top three for four questions out of six. It
+is now BM25 over weighted fields, where a word's worth comes from how rare it
+is in *your* corpus and a word in a declaration's name outweighs the same word
+buried in a body.
+
+Twenty-one of the thirty-five still fail, and the largest remaining cause is
+named in [docs/TESTING.md](docs/TESTING.md): a test declaration often outranks
+the code it tests, because it repeats that code's vocabulary and adds its own.
+
 **What this is:** it moves the semantic work from query time to index time.
 Matching stays lexical. It is LLM-authored keyword expansion, and its reach is
 bounded by how many ways of saying the thing the agent thought to write down.

{rag_your_code-0.5.0 → rag_your_code-0.6.0}/src/rag_your_code.egg-info/SOURCES.txt

@@ -36,6 +36,7 @@ tests/test_metadata.py
 tests/test_multilanguage.py
 tests/test_parser_edges.py
 tests/test_ragyourcode.py
+tests/test_ranking.py
 tests/test_repo_queries.py
 tests/test_resilience.py
 tests/test_retrieval_correctness.py

{rag_your_code-0.5.0 → rag_your_code-0.6.0}/src/ragyourcode/__init__.py

@@ -3,4 +3,4 @@
 from .models import CodeUnit, SearchResult
 
 __all__ = ["CodeUnit", "SearchResult"]
-__version__ = "0.5.0"
+__version__ = "0.6.0"

{rag_your_code-0.5.0 → rag_your_code-0.6.0}/src/ragyourcode/cli.py

@@ -18,7 +18,7 @@ from .document import plan as plan_documentation, render_patch, summarise as sum
 from .embeddings import embed, embedding_metadata
 from .graph import build_graph, graph_from_dict, graph_search
 from .indexer import StaleMonitor, build_fingerprint, build_units, fingerprint, index_build_fingerprint, read_index, snapshot_repository, write_index
-from .search import build_search_index, context, search
+from .search import build_search_index, context, search, within_budget
 
 # Derived from the settings table so the default is written down once.
 # `tests/test_agent_protocol.py` imports these to assert the bound it enforces.
@@ -157,7 +157,10 @@ def _cmd_search(args: argparse.Namespace) -> int:
         else search(units, args.query, limit, search_index=search_index, vector_weight=weight)
     )
     if args.json:
-        print(json.dumps({"query": args.query, "mode": "graph" if args.graph else "hybrid", "stale": payload.get("stale", True), "degraded": payload.get("degraded"), "results": [result.to_dict() for result in results], "context": context(results, max_chars)}, ensure_ascii=False))
+        # One budget decision, applied once: the results an agent reads and the
+        # context beside them are the same set, so they cannot disagree.
+        shown = within_budget(results, max_chars)
+        print(json.dumps({"query": args.query, "mode": "graph" if args.graph else "hybrid", "stale": payload.get("stale", True), "degraded": payload.get("degraded"), "results": [result.to_dict() for result in shown], "omitted_for_budget": len(results) - len(shown), "context": context(shown, max_chars)}, ensure_ascii=False))
     else:
         if payload.get("stale"):
             print("Warning: index is stale; run `rag-your-code index` to refresh.", file=sys.stderr)
@@ -496,7 +499,9 @@ def _cmd_agent(args: argparse.Namespace) -> int:
                     if use_graph
                     else search(units, query, limit, search_index=search_index, vector_weight=weight)
                 )
-                response = {"stale": payload.get("stale", True), "results": [result.to_dict() for result in results], "context": context(results, _request_int(request, "max_chars", default_chars, 0, 100000))}
+                budget = _request_int(request, "max_chars", default_chars, 0, 100000)
+                shown = within_budget(results, budget)
+                response = {"stale": payload.get("stale", True), "results": [result.to_dict() for result in shown], "omitted_for_budget": len(results) - len(shown), "context": context(shown, budget)}
             elif action == "research":
                 response = research(
                     units,

{rag_your_code-0.5.0 → rag_your_code-0.6.0}/src/ragyourcode/models.py

@@ -28,24 +28,38 @@ class CodeUnit:
     imports: list[str] = field(default_factory=list)
     vector: Sequence[float] = field(default_factory=list)
 
+    @property
+    def searchable_fields(self) -> dict[str, str]:
+        """Everything retrieval may match, kept apart by where the author
+        wrote it. Where a word appears is itself evidence of how much it
+        means: a term in a declaration's name is what the author decided to
+        call the thing, while the same term two hundred lines into a body is
+        a passing mention. Ranking weights those differently, so they cannot
+        arrive as one undifferentiated blob -- flattened, a long function
+        outranks the function that is actually named after the query, purely
+        by owning more words.
+
+        Because the description is one of these fields, replacing a generated
+        description with a better one immediately widens the set of queries
+        that can reach this unit.
+        """
+        return {
+            "name": f"{self.qualified_name} {self.kind}",
+            "signature": self.signature,
+            "description": self.description,
+            "relations": "calls: " + " ".join(self.calls) + "\nimports: " + " ".join(self.imports),
+            "body": self.source,
+        }
+
     @property
     def searchable_text(self) -> str:
-        """Assembles everything about a unit that retrieval is allowed to match
-        against: qualified name, kind and signature, the description, the
-        names it calls and imports, and the full source. Because the
-        description is part of this text, replacing a generated description
-        with a better one immediately widens the set of queries that can
-        reach this unit.
+        """Every searchable field as one block, for callers that want the
+        words without caring where they came from -- the unit's own embedding
+        is built from this. Derived from ``searchable_fields`` rather than
+        rebuilt beside it, so a field added for ranking cannot go missing
+        from the text that gets embedded.
         """
-        return "\n".join(
-            (
-                f"{self.qualified_name} {self.kind} {self.signature}",
-                self.description,
-                "calls: " + " ".join(self.calls),
-                "imports: " + " ".join(self.imports),
-                self.source,
-            )
-        )
+        return "\n".join(self.searchable_fields.values())
 
     def to_dict(self, include_vector: bool = True) -> dict[str, Any]:
         """Serialises a unit to a plain dictionary for storage or for an agent

rag_your_code-0.6.0/src/ragyourcode/search.py

@@ -0,0 +1,278 @@
+"""Hybrid lexical/vector retrieval for agent context."""
+
+from __future__ import annotations
+
+import heapq
+import math
+from bisect import bisect_left
+from collections import Counter, defaultdict
+from dataclasses import dataclass
+
+from .config import BY_PATH
+from .embeddings import DEFAULT_DIMENSIONS, embed, tokenize
+from .models import CodeUnit, SearchResult
+
+# Named here rather than repeated as a literal so `search.vector_weight` in
+# rag-your-code.toml and the default a direct caller gets cannot drift apart.
+DEFAULT_VECTOR_WEIGHT: float = BY_PATH["search.vector_weight"].default
+
+# How much a word counts for, by the field the author wrote it in. A term in
+# the name is what the declaration is called; the same term inside the body is
+# a mention. Every key of `CodeUnit.searchable_fields` must appear here, and a
+# test asserts the two sets agree -- a field with no weight would otherwise
+# vanish from ranking the moment it was added, silently.
+FIELD_WEIGHTS: dict[str, float] = {
+    "name": 8.0,
+    "signature": 4.0,
+    "description": 3.0,
+    "relations": 2.0,
+    "body": 1.0,
+}
+
+# BM25's saturation and length-normalisation constants, at their standard
+# values. `k1` bounds what repeating a word can buy; `b` decides how hard a
+# field is discounted for being longer than its average.
+#
+# Three variations were implemented and measured against all three rulers
+# before settling here, and two were dropped for want of evidence: excluding
+# curated text from the length on the argument that a written description is
+# deliberate rather than incidental, and counting authored words instead of
+# tokeniser output so a run of Chinese expanded into overlapping bigrams would
+# not read as five times the text. Each moved one to four questions in both
+# directions at once, which is this instrument's noise. Lowering `b` to 0.5 or
+# 0.3 under per-field normalisation was measured and was worse than 0.75 on
+# the foreign-repository ruler. See docs/TESTING.md.
+BM25_K1 = 1.2
+BM25_B = 0.75
+
+
+@dataclass(slots=True)
+class SearchIndex:
+    """In-memory inverted index reused across queries.
+
+    Building this once avoids re-tokenizing every code unit. Each posting
+    carries a term's weight in that unit alongside its id -- already scaled by
+    the field the term appeared in and by how long that field is -- because
+    ranking needs to know how often a term occurs and where, not merely that
+    it occurs somewhere. The weight lives in the posting rather than in a
+    per-unit table: an earlier version cached a per-unit frozenset of every
+    token, which cost the largest share of resident memory while holding
+    nothing the postings did not already have.
+
+    Lists are kept sorted by unit id so membership can be answered by binary
+    search. The same structure can later be backed by SQLite/ANN storage.
+    """
+
+    units: dict[str, CodeUnit]
+    postings: dict[str, tuple[tuple[str, float], ...]]
+
+
+def build_search_index(units: list[CodeUnit]) -> SearchIndex:
+    """Builds the inverted lookup table, recording for each term the units that
+    contain it and how much it counts for in each.
+
+    A term's weight is BM25F's: its count in a field, divided by how long that
+    field is against the average for that same field, then scaled by what the
+    field is worth. Normalising per field is the part that matters. Measured
+    against one length for the whole unit, a body repeating a word forty times
+    still beat the declaration actually named after it, because a long body's
+    advantage in raw count almost exactly cancelled its penalty for being
+    long. Comparing each field against its own average removes that cancelling.
+
+    Two passes are needed because a field's average length is a property of
+    the corpus and is not known until every unit has been seen. The second
+    pass re-tokenizes rather than holding every unit's tokens in memory at
+    once, which costs about half again in time and keeps the peak bounded.
+    """
+    field_totals: Counter[str] = Counter()
+    for unit in units:
+        for field_name, text in unit.searchable_fields.items():
+            field_totals[field_name] += len(tokenize(text))
+    count = len(units) or 1
+    # A field every unit leaves empty would otherwise divide by zero. Its terms
+    # cannot reach any unit anyway, so the value only has to be finite.
+    averages = {name: (total / count) or 1.0 for name, total in field_totals.items()}
+
+    postings: dict[str, list[tuple[str, float]]] = defaultdict(list)
+    for unit in units:
+        weighted: Counter[str] = Counter()
+        for field_name, text in unit.searchable_fields.items():
+            terms = tokenize(text)
+            if not terms:
+                continue
+            saturation = 1 - BM25_B + BM25_B * len(terms) / averages[field_name]
+            weight = FIELD_WEIGHTS[field_name] / saturation
+            for term, occurrences in Counter(terms).items():
+                weighted[term] += weight * occurrences
+        for term, mass in weighted.items():
+            postings[term].append((unit.id, mass))
+    return SearchIndex(
+        {unit.id: unit for unit in units},
+        {term: tuple(sorted(entries)) for term, entries in postings.items()},
+    )
+
+
+def _in_posting(posting: tuple[tuple[str, float], ...], unit_id: str) -> bool:
+    """Membership test over a posting list, which build_search_index keeps
+    sorted by unit id. A one-element tuple sorts before every pair sharing its
+    id, so it locates the entry without needing the mass that follows it.
+    """
+    position = bisect_left(posting, (unit_id,))
+    return position < len(posting) and posting[position][0] == unit_id
+
+
+def _inverse_document_frequency(document_count: int, matching: int) -> float:
+    """How much evidence one term carries, from how rare it is in this corpus.
+
+    A word in nearly every unit says nothing about which unit is wanted, and
+    a word in two says a great deal. Deriving that from the corpus rather than
+    from a list of stopwords is what makes it work on a repository in any
+    language: `the` and `calls` earn their low weight the same way a Chinese
+    bigram does, by being everywhere, and no list has to be maintained.
+    """
+    return math.log(1 + (document_count - matching + 0.5) / (matching + 0.5))
+
+
+def search(
+    units: list[CodeUnit],
+    query: str,
+    limit: int = 8,
+    search_index: SearchIndex | None = None,
+    vector_weight: float = DEFAULT_VECTOR_WEIGHT,
+) -> list[SearchResult]:
+    """Ranks code units against a natural-language query by combining weighted
+    word overlap with vector similarity.
+
+    The lexical half is BM25 over weighted fields: rare words count for more
+    than common ones, repeating a word saturates rather than accumulating
+    without bound, and a unit is normalised by how much text it owns. An
+    earlier version scored the plain fraction of query words present, which
+    made `the` worth as much as `daemon` and handed every query to whichever
+    unit was longest -- on a foreign repository the single largest declaration
+    came back for four questions out of six.
+
+    Every unit sharing any query word is scored, so nothing that matches is
+    left out. Full vector scoring is reserved for units reached by a selective
+    term, since computing a dot product for everything a stopword-class term
+    touches is pure cost. With no overlap anywhere it falls back to similarity
+    alone.
+    """
+    query_tokens = set(tokenize(query))
+    if limit <= 0 or not query_tokens:
+        return []
+    query_vector = embed(query, len(units[0].vector) if units and units[0].vector else DEFAULT_DIMENSIONS)
+    query_features = [(index, value) for index, value in enumerate(query_vector) if value]
+    search_index = search_index or build_search_index(units)
+    postings = [(token, search_index.postings.get(token, ())) for token in query_tokens]
+    document_count = len(search_index.units) or 1
+
+    # Accumulate term by term rather than unit by unit: a posting list is
+    # exactly the units a term reaches, so this touches no unit the query
+    # cannot possibly match. Length normalisation is already inside `mass`, so
+    # all that remains is saturation -- the tenth occurrence of a word says
+    # much less than the second, and neither should be able to run away with
+    # the ranking.
+    weights = {token: _inverse_document_frequency(document_count, len(posting)) for token, posting in postings if posting}
+    lexical_scores: dict[str, float] = defaultdict(float)
+    for token, posting in postings:
+        if not posting:
+            continue
+        weight = weights[token]
+        for unit_id, mass in posting:
+            lexical_scores[unit_id] += weight * mass * (BM25_K1 + 1) / (mass + BM25_K1)
+
+    # Divide by what this query could have scored at most, which is a constant
+    # for the query and so changes no ranking. It exists to keep the lexical
+    # half on the 0..1 scale `search.vector_weight` was documented and bounded
+    # against: raw BM25 is unbounded, and against a score of 18 a weight of
+    # 0.35 on a cosine would be arithmetic, not a tie-break.
+    ceiling = sum(weights.values()) * (BM25_K1 + 1)
+
+    # A term present in a tenth of the corpus (``function``, ``return``) is not
+    # evidence of relevance, and its posting list is effectively the whole index;
+    # computing a 384-dimension dot product for everything it reaches is what
+    # this threshold exists to avoid. It selects which candidates additionally
+    # receive a VECTOR score. It must not decide which candidates are scored at
+    # all -- doing that silently dropped units matching MORE query terms and
+    # under-filled ``limit`` (116 units, `--limit 8`, one result returned).
+    selective_threshold = max(64, min(2048, len(units) // 10))
+    vector_ids: set[str] = set()
+    for _, posting in postings:
+        if 0 < len(posting) <= selective_threshold:
+            vector_ids.update(unit_id for unit_id, _ in posting)
+    if lexical_scores and not vector_ids and len(lexical_scores) <= selective_threshold:
+        vector_ids = set(lexical_scores)
+
+    # With no lexical overlap anywhere, fall back to pure cosine so a genuine
+    # paraphrase still retrieves something.
+    candidate_ids = lexical_scores.keys() if lexical_scores else search_index.units.keys()
+    scored: list[tuple[float, str]] = []
+    for unit_id in candidate_ids:
+        unit = search_index.units[unit_id]
+        lexical = (lexical_scores.get(unit_id, 0.0) / ceiling) if ceiling else 0.0
+        vector_score = (
+            sum(value * unit.vector[index] for index, value in query_features)
+            if (unit_id in vector_ids or not lexical_scores) and len(unit.vector) == len(query_vector)
+            else 0.0
+        )
+        # Exact symbols and domain terms are high-confidence evidence. Keep
+        # lexical overlap dominant so a noisy feature-hash vector cannot push an
+        # exact match below an unrelated semantic neighbor; use the vector score
+        # to rank paraphrases and break lexical ties.
+        score = lexical + vector_weight * max(0.0, vector_score)
+        if lexical or score > 0:
+            scored.append((score, unit_id))
+    # Materialise only the winners. Building a SearchResult for every lexical
+    # match and then sorting all of them cost more than the scoring itself once
+    # recall became complete: at 10k units that alone was most of a 10x query
+    # regression. nsmallest keeps the exact previous ordering -- highest score
+    # first, ties broken by ascending unit id -- at O(n log limit).
+    winners = heapq.nsmallest(limit, scored, key=lambda item: (-item[0], item[1]))
+    # Which terms matched is only needed for the handful actually returned, and
+    # postings are stored sorted, so a binary search beats carrying a per-unit
+    # term list through the scoring loop for every candidate in the corpus.
+    return [
+        SearchResult(search_index.units[unit_id], score, sorted(token for token, posting in postings if _in_posting(posting, unit_id)))
+        for score, unit_id in winners
+    ]
+
+
+def _block(result: SearchResult) -> str:
+    """One result as an agent reads it: identifier, score, why it matched,
+    what it is, and the code itself.
+    """
+    unit = result.unit
+    evidence = "\nEvidence: " + " | ".join(result.evidence) if result.evidence else ""
+    return f"[{unit.id}] score={result.score:.3f}{evidence}\n{unit.description}\n```{unit.language}\n{unit.source}\n```"
+
+
+def within_budget(results: list[SearchResult], max_chars: int) -> list[SearchResult]:
+    """The leading results whose rendered size fits the caller's budget.
+
+    The budget has to decide how many results there *are*, not merely how many
+    get rendered into one of the two places they are sent. `search --json`
+    used to serialise every result in full while capping only the context
+    string beside them, so a default query answered with 65,025 characters
+    against a stated budget of 12,000 -- and the agent, which reads the
+    results, was the side that overran.
+
+    The first result is always kept. A search that found something and
+    returned nothing because the match was large is less useful than one
+    oversized answer, and the caller is told how many were dropped.
+    """
+    kept: list[SearchResult] = []
+    used = 0
+    for result in results:
+        size = len(_block(result))
+        if kept and used + size > max_chars:
+            break
+        kept.append(result)
+        used += size
+    return kept
+
+
+def context(results: list[SearchResult], max_chars: int = 12000) -> str:
+    """Packs ranked results into one readable block for an agent prompt,
+    stopping before the caller's character budget is exceeded.
+    """
+    return "\n\n".join(_block(result) for result in within_budget(results, max_chars))

rag_your_code-0.6.0/tests/test_ranking.py

@@ -0,0 +1,202 @@
+"""Guards for the three ranking defects a cold foreign repository exposed.
+
+Measured on cc-enforcer -- 1153 units, no written descriptions -- the single
+largest declaration came back in the top three for four questions out of six,
+and the words deciding the ranking were the ones present in nearly every unit.
+Three causes, all in how a match was scored:
+
+H: every query word counted the same, so `the` (49% of units) and `calls`
+   (97%) outweighed `daemon` (2 units) and `warm` (none).
+I: nothing normalised for size, and the largest unit held 539 distinct terms
+   against a median of 52, so it could contain any query by accident.
+J: a term in a declaration's name counted no more than the same term two
+   hundred lines into a body, which put test units above the code they test.
+
+Each test is built so that the previous rule -- the fraction of query words
+present anywhere in the unit -- ranks the *wrong* answer first.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+from ragyourcode.indexer import build_units
+from ragyourcode.models import CodeUnit
+from ragyourcode.search import FIELD_WEIGHTS, _block, build_search_index, context, search, within_budget
+
+RARE_QUERY = "handle the request and return the response quiesce"
+
+
+def _with_a_word_nothing_else_uses(root: Path) -> Path:
+    """One more unit, whose only distinguishing word appears nowhere else."""
+    (root / "rare.py").write_text(
+        "def quiesce(attempt):\n"
+        '    """Settle down."""\n'
+        "    return attempt\n",
+        encoding="utf-8",
+    )
+    return root
+
+
+def test_a_word_in_almost_every_unit_cannot_decide_the_ranking(tmp_path: Path, skewed_corpus):
+    """Rarity is evidence; ubiquity is not.
+
+    Under the previous rule the unit matching six common words out of seven
+    scored 0.857 and the only unit that answers the question scored 0.286.
+    Nothing here is a stopword list: `the` and `request` earn their low weight
+    from this corpus, which is why the mechanism works on a repository written
+    in any language, including one where the tokens are Chinese bigrams.
+    """
+    units = build_units(_with_a_word_nothing_else_uses(skewed_corpus(tmp_path)))
+    results = search(units, RARE_QUERY, limit=3)
+    assert results[0].unit.name == "quiesce", [result.unit.id for result in results]
+
+
+def test_matched_terms_survive_the_posting_list_lookup(tmp_path: Path, skewed_corpus):
+    """Postings now carry a weight beside each id, and are searched by id.
+
+    A membership test comparing whole entries would report no matched terms at
+    all, turning every result's evidence -- the thing that makes a result
+    checkable -- into an empty list.
+    """
+    units = build_units(_with_a_word_nothing_else_uses(skewed_corpus(tmp_path)))
+    results = search(units, RARE_QUERY, limit=8)
+    assert all(result.matched_terms for result in results), [
+        (result.unit.id, result.matched_terms) for result in results
+    ]
+
+
+def test_a_long_unit_does_not_win_on_size_alone(tmp_path: Path, skewed_corpus):
+    """Containing a word is not the same as being about it.
+
+    The corpus has to be this large for the defect to appear at all. Every
+    query word here reaches all eighty-one units, which puts its posting list
+    past the selective threshold, so no vector score is computed and the
+    lexical rule decides alone. In a two-unit fixture the feature-hash vector
+    hides the problem, because normalising by a unit's token mass is itself a
+    crude length normalisation -- which is why the defect was visible on a
+    1153-unit repository and invisible in the small benchmark.
+    """
+    filler = "\n".join(f"    step_{number} = compute(step_{number - 1})" for number in range(1, 200))
+    (skewed_corpus(tmp_path) / "aggregate.py").write_text(
+        "def process_everything(request, response):\n"
+        '    """Handle request and return response."""\n'
+        f"{filler}\n"
+        "    return response\n",
+        encoding="utf-8",
+    )
+    units = build_units(tmp_path)
+    results = search(units, "handle the request and return the response", limit=3)
+    # Same words as every handler, two hundred lines of unrelated code, and it
+    # came back first: matched-word count made them equal and the tie fell to
+    # the unit id.
+    assert "process_everything" not in [result.unit.name for result in results], [
+        result.unit.id for result in results
+    ]
+
+
+def test_the_same_word_counts_for_more_in_a_name_than_in_a_body(tmp_path: Path):
+    """Where the author wrote a word says how much it meant.
+
+    Both units are the same size and mention the word once, so nothing but the
+    field it appears in can separate them. The vector is switched off because
+    it would answer this question by a different route -- its cosine already
+    favours the unit where the word is a larger share of the text -- and the
+    lexical rule is what changed. Without that, both scored exactly 1.0 and
+    the tie fell to the unit id.
+
+    This is the mechanism, not its limit: a body repeating a word forty times
+    still outranks the declaration named after it, because forty occurrences
+    over an average-length body is genuinely a lot of evidence. See
+    docs/TESTING.md for what that still costs.
+    """
+    (tmp_path / "named.py").write_text(
+        'def checkpoint(job):\n    """Do the work and hand it back."""\n    return job\n',
+        encoding="utf-8",
+    )
+    (tmp_path / "mentions.py").write_text(
+        'def worker(job):\n    """Do the work and hand it back."""\n    # checkpoint\n    return job\n',
+        encoding="utf-8",
+    )
+    units = build_units(tmp_path)
+    results = search(units, "checkpoint", limit=2, vector_weight=0.0)
+    assert results[0].unit.name == "checkpoint", [result.unit.id for result in results]
+
+
+def test_every_searchable_field_carries_a_weight(tmp_path: Path):
+    """A field with no weight would vanish from ranking the moment it appeared.
+
+    The two live in different modules on purpose -- one says what retrieval may
+    match, the other how much each part counts -- so this is the join that
+    keeps them from drifting apart.
+    """
+    (tmp_path / "one.py").write_text("def f(a):\n    return a\n", encoding="utf-8")
+    unit = build_units(tmp_path)[0]
+    assert set(unit.searchable_fields) == set(FIELD_WEIGHTS)
+
+
+def test_searchable_text_is_every_field_and_nothing_else(tmp_path: Path):
+    """The text a unit is embedded from must not lose a field ranking uses."""
+    (tmp_path / "one.py").write_text(
+        '"""Module."""\n\n\ndef f(a):\n    """Add one."""\n    return a + 1\n',
+        encoding="utf-8",
+    )
+    unit = build_units(tmp_path)[0]
+    assert unit.searchable_text == "\n".join(unit.searchable_fields.values())
+
+
+def _budgeted(root: Path) -> list:
+    for index in range(6):
+        (root / f"unit_{index}.py").write_text(
+            f"def widget_{index}(value):\n"
+            f'    """Widget {index} handles the value."""\n'
+            + "".join(f"    line_{number} = value\n" for number in range(40))
+            + "    return value\n",
+            encoding="utf-8",
+        )
+    return search(build_units(root), "widget handles the value", limit=6)
+
+
+def test_the_budget_bounds_the_results_not_only_the_context(tmp_path: Path):
+    """`search --json` served 65,025 characters against a budget of 12,000.
+
+    The cap applied to the context string while every result was serialised
+    beside it in full, so the half an agent reads was the half that overran.
+    """
+    results = _budgeted(tmp_path)
+    assert len(results) > 1, "the fixture must produce enough results to be trimmed"
+    budget = len(_block(results[0])) + 10
+    kept = within_budget(results, budget)
+    assert len(kept) < len(results)
+    assert sum(len(_block(result)) for result in kept) <= budget
+    assert context(kept, budget) == context(results, budget)
+
+
+def test_a_single_result_larger_than_the_budget_is_still_returned(tmp_path: Path):
+    """Finding something and returning nothing is worse than one oversized hit."""
+    results = _budgeted(tmp_path)
+    assert [result.unit.id for result in within_budget(results, 0)] == [results[0].unit.id]
+
+
+def test_a_field_every_unit_leaves_empty_does_not_divide_by_zero():
+    """Each field is normalised against that field's own average length.
+
+    A corpus where nobody fills one in -- no signature, no body, no written
+    description -- makes that average zero, and the average is a denominator.
+    """
+    unit = CodeUnit(
+        id="empty.py:1:nothing",
+        path="empty.py",
+        language="python",
+        kind="function",
+        name="nothing",
+        qualified_name="nothing",
+        signature="",
+        start_line=1,
+        end_line=1,
+        source="",
+        description="",
+        serial=1,
+    )
+    index = build_search_index([unit])
+    assert search([unit], "nothing", limit=1, search_index=index)[0].unit.id == unit.id

{rag_your_code-0.5.0 → rag_your_code-0.6.0}/tests/test_repo_queries.py

@@ -21,30 +21,42 @@ from ragyourcode import descriptions as descriptions_module
 from ragyourcode.indexer import build_units
 
 ROOT = Path(__file__).resolve().parents[1]
+COLD_PATH = ROOT / "benchmarks" / "cold_queries.json"
 
 
 @pytest.fixture(scope="module")
 def units():
     # With the description store, because that is how the CLI builds an index.
-    # Without it the same ruler scores 0.171 rather than 0.500 on hit@1, which
-    # would be measuring a configuration nobody runs.
     return build_units(ROOT, descriptions=descriptions_module.load(ROOT))
 
 
+@pytest.fixture(scope="module")
+def cold_units():
+    """The same repository as a first-time user's index sees it: parsed, with
+    only the sentence the parser generates and nothing anybody wrote.
+    """
+    return build_units(ROOT)
+
+
 def test_every_acceptable_answer_names_code_that_exists(units):
     problems = check_ruler(load_questions(), units)
     assert not problems, "the ruler has drifted from the code:\n  " + "\n  ".join(problems)
 
 
-def test_the_ruler_is_well_formed():
-    questions = load_questions()
+@pytest.mark.parametrize("path", [QUERIES_PATH, COLD_PATH], ids=["repository", "cold"])
+def test_the_ruler_is_well_formed(path: Path):
+    questions = load_questions(path)
     entries = questions["queries"]
-    assert len(entries) >= 50, "too few questions to distinguish a change from noise"
+    assert len(entries) >= 30, "too few questions to distinguish a change from noise"
     assert questions["k"] >= 1
+    seen: set[str] = set()
     for entry in entries:
         assert entry["query"].strip(), f"{entry['id']}: empty question"
+        assert entry["id"] not in seen, f"{entry['id']}: duplicate question id"
+        seen.add(entry["id"])
         assert entry["language"] in {"en", "zh"}
         assert entry["kind"] in {"concept", "why", "symbol"}
+        assert entry["acceptable"], f"{entry['id']}: no acceptable answer listed"
         assert all(len(pair) == 2 for pair in entry["acceptable"])
     # Both languages have to be represented, because the CJK path through the
     # tokenizer is the one place a query can share no character class with the
@@ -53,6 +65,29 @@ def test_the_ruler_is_well_formed():
     assert languages == {"en", "zh"}
 
 
+def test_the_cold_ruler_says_which_repository_it_grades():
+    """It grades code that is not in this repository, so it has to name it and
+    say why a ruler asked about this project cannot stand in for it.
+    """
+    questions = load_questions(COLD_PATH)
+    assert questions["repository"], "a ruler over foreign code must name that code"
+    assert questions["caveat"].strip()
+    assert questions["why"].strip()
+
+
+def test_written_descriptions_beat_generated_ones(units, cold_units):
+    """The reason `describe` exists, asserted rather than stated.
+
+    This used to be a comment quoting two numbers, which is exactly the kind of
+    claim that rots: both had already moved by the time anybody looked.
+    """
+    questions = load_questions()
+    warm = evaluate(units, questions)["aggregate"]
+    cold = evaluate(cold_units, questions)["aggregate"]
+    assert warm["hit_at_1"] > cold["hit_at_1"], (warm, cold)
+    assert warm["mrr"] > cold["mrr"], (warm, cold)
+
+
 def test_the_ruler_has_headroom(units):
     """A ruler everything already passes cannot measure an improvement.
 

{rag_your_code-0.5.0 → rag_your_code-0.6.0}/tests/test_retrieval_correctness.py

@@ -17,31 +17,13 @@ from ragyourcode.indexer import build_units, fingerprint, read_index, snapshot_r
 from ragyourcode.search import build_search_index, search
 
 
-def _corpus(tmp_path: Path, count: int = 80) -> Path:
-    """More units than the selective threshold's 64 floor, with skewed terms."""
-    for index in range(count):
-        (tmp_path / f"mod_{index}.py").write_text(
-            f"def handler_{index}(request, response):\n"
-            f'    """Handle request and return response."""\n'
-            f"    return response\n",
-            encoding="utf-8",
-        )
-    (tmp_path / "special.py").write_text(
-        "def retry_request_with_backoff(request, response):\n"
-        '    """Retry a failed request and return response."""\n'
-        "    return response\n",
-        encoding="utf-8",
-    )
-    return tmp_path
-
-
-def test_limit_is_filled_when_a_rare_term_joins_common_ones(tmp_path: Path):
+def test_limit_is_filled_when_a_rare_term_joins_common_ones(tmp_path: Path, skewed_corpus):
     """A rare token must not shrink the candidate set to itself.
 
     `backoff` reaches one unit; `request` and `response` reach all 81. Scoring
     only what the rare term reached returned a single result for `--limit 8`.
     """
-    units = build_units(_corpus(tmp_path))
+    units = build_units(skewed_corpus(tmp_path))
     assert len(units) > 64, "the corpus must exceed the selective threshold's floor"
     index = build_search_index(units)
     results = search(units, "backoff request response", limit=8, search_index=index)
@@ -50,8 +32,8 @@ def test_limit_is_filled_when_a_rare_term_joins_common_ones(tmp_path: Path):
     assert all(result.matched_terms for result in results), "every returned unit must carry its evidence"
 
 
-def test_every_lexically_matching_unit_can_be_returned(tmp_path: Path):
-    units = build_units(_corpus(tmp_path, count=70))
+def test_every_lexically_matching_unit_can_be_returned(tmp_path: Path, skewed_corpus):
+    units = build_units(skewed_corpus(tmp_path, count=70))
     index = build_search_index(units)
     results = search(units, "handler_7 request", limit=100, search_index=index)
     returned = {result.unit.id for result in results}

rag_your_code-0.5.0/src/ragyourcode/search.py

@@ -1,151 +0,0 @@
-"""Hybrid lexical/vector retrieval for agent context."""
-
-from __future__ import annotations
-
-import heapq
-from bisect import bisect_left
-from collections import Counter, defaultdict
-from dataclasses import dataclass
-
-from .config import BY_PATH
-from .embeddings import DEFAULT_DIMENSIONS, embed, tokenize
-from .models import CodeUnit, SearchResult
-
-# Named here rather than repeated as a literal so `search.vector_weight` in
-# rag-your-code.toml and the default a direct caller gets cannot drift apart.
-DEFAULT_VECTOR_WEIGHT: float = BY_PATH["search.vector_weight"].default
-
-
-@dataclass(slots=True)
-class SearchIndex:
-    """In-memory inverted index reused across queries.
-
-    Building this once avoids re-tokenizing every code unit. Matched terms are
-    read straight out of ``postings``; an earlier version also cached a
-    per-unit frozenset of every token, which cost the largest share of the
-    index's resident memory while holding nothing ``postings`` did not already
-    have. The same structure can later be backed by SQLite/ANN storage.
-    """
-
-    units: dict[str, CodeUnit]
-    postings: dict[str, tuple[str, ...]]
-
-
-def build_search_index(units: list[CodeUnit]) -> SearchIndex:
-    """Builds the inverted lookup table by tokenising every unit once and
-    recording, for each term, which units contain it. The lists are kept
-    sorted so membership can later be answered by binary search rather than
-    by carrying a word set through the scoring loop.
-    """
-    postings: dict[str, set[str]] = defaultdict(set)
-    for unit in units:
-        for term in set(tokenize(unit.searchable_text)):
-            postings[term].add(unit.id)
-    return SearchIndex({unit.id: unit for unit in units}, {term: tuple(sorted(ids)) for term, ids in postings.items()})
-
-
-def _in_posting(posting: tuple[str, ...], unit_id: str) -> bool:
-    """Membership test over a posting list, which build_search_index keeps sorted."""
-    position = bisect_left(posting, unit_id)
-    return position < len(posting) and posting[position] == unit_id
-
-
-def search(
-    units: list[CodeUnit],
-    query: str,
-    limit: int = 8,
-    search_index: SearchIndex | None = None,
-    vector_weight: float = DEFAULT_VECTOR_WEIGHT,
-) -> list[SearchResult]:
-    """Ranks code units against a natural-language query by combining word
-    overlap with vector similarity. Every unit sharing any query word is
-    scored, so nothing that matches is left out: an earlier design let the
-    vector shortlist decide who was scored at all, and units matching more
-    query words went unranked, returning one result where eight were asked
-    for. Full vector scoring is reserved for units reached by a selective
-    term, since computing a dot product for everything a stopword-class term
-    touches is pure cost. Word overlap stays dominant so an exact symbol
-    match cannot be pushed below a noisy neighbour, and with no overlap
-    anywhere it falls back to similarity alone.
-    """
-    query_tokens = set(tokenize(query))
-    if limit <= 0 or not query_tokens:
-        return []
-    query_vector = embed(query, len(units[0].vector) if units and units[0].vector else DEFAULT_DIMENSIONS)
-    query_features = [(index, value) for index, value in enumerate(query_vector) if value]
-    search_index = search_index or build_search_index(units)
-    postings = [(token, search_index.postings.get(token, ())) for token in query_tokens]
-
-    # Matched terms come straight from the posting lists. Walking postings costs
-    # O(sum of posting lengths) of dict work, where scoring each candidate by
-    # intersecting a cached per-unit token set cost a frozenset operation per
-    # candidate -- and every lexically matching unit now gets a score.
-    matched_counts: Counter[str] = Counter()
-    for _, posting in postings:
-        matched_counts.update(posting)
-
-    # A term present in a tenth of the corpus (``function``, ``return``) is not
-    # evidence of relevance, and its posting list is effectively the whole index;
-    # computing a 384-dimension dot product for everything it reaches is what
-    # this threshold exists to avoid. It selects which candidates additionally
-    # receive a VECTOR score. It must not decide which candidates are scored at
-    # all -- doing that silently dropped units matching MORE query terms and
-    # under-filled ``limit`` (116 units, `--limit 8`, one result returned).
-    selective_threshold = max(64, min(2048, len(units) // 10))
-    vector_ids: set[str] = set()
-    for _, posting in postings:
-        if 0 < len(posting) <= selective_threshold:
-            vector_ids.update(posting)
-    if matched_counts and not vector_ids and len(matched_counts) <= selective_threshold:
-        vector_ids = set(matched_counts)
-
-    # With no lexical overlap anywhere, fall back to pure cosine so a genuine
-    # paraphrase still retrieves something.
-    candidate_ids = matched_counts.keys() if matched_counts else search_index.units.keys()
-    scored: list[tuple[float, str]] = []
-    for unit_id in candidate_ids:
-        unit = search_index.units[unit_id]
-        lexical = matched_counts.get(unit_id, 0) / len(query_tokens)
-        vector_score = (
-            sum(value * unit.vector[index] for index, value in query_features)
-            if (unit_id in vector_ids or not matched_counts) and len(unit.vector) == len(query_vector)
-            else 0.0
-        )
-        # Exact symbols and domain terms are high-confidence evidence. Keep
-        # lexical overlap dominant so a noisy feature-hash vector cannot push an
-        # exact match below an unrelated semantic neighbor; use the vector score
-        # to rank paraphrases and break lexical ties.
-        score = lexical + vector_weight * max(0.0, vector_score)
-        if lexical or score > 0:
-            scored.append((score, unit_id))
-    # Materialise only the winners. Building a SearchResult for every lexical
-    # match and then sorting all of them cost more than the scoring itself once
-    # recall became complete: at 10k units that alone was most of a 10x query
-    # regression. nsmallest keeps the exact previous ordering -- highest score
-    # first, ties broken by ascending unit id -- at O(n log limit).
-    winners = heapq.nsmallest(limit, scored, key=lambda item: (-item[0], item[1]))
-    # Which terms matched is only needed for the handful actually returned, and
-    # postings are stored sorted, so a binary search beats carrying a per-unit
-    # term list through the scoring loop for every candidate in the corpus.
-    return [
-        SearchResult(search_index.units[unit_id], score, sorted(token for token, posting in postings if _in_posting(posting, unit_id)))
-        for score, unit_id in winners
-    ]
-
-
-def context(results: list[SearchResult], max_chars: int = 12000) -> str:
-    """Packs ranked results into one readable block for an agent prompt, each
-    carrying identifier, score, matching evidence, description and source,
-    and stops before exceeding the caller character budget.
-    """
-    blocks: list[str] = []
-    used = 0
-    for result in results:
-        unit = result.unit
-        evidence = "\nEvidence: " + " | ".join(result.evidence) if result.evidence else ""
-        block = f"[{unit.id}] score={result.score:.3f}{evidence}\n{unit.description}\n```{unit.language}\n{unit.source}\n```"
-        if used + len(block) > max_chars:
-            break
-        blocks.append(block)
-        used += len(block)
-    return "\n\n".join(blocks)