rag-your-code 0.6.0__tar.gz → 0.7.0__tar.gz

{rag_your_code-0.6.0/src/rag_your_code.egg-info → rag_your_code-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rag-your-code
-Version: 0.6.0
+Version: 0.7.0
 Summary: A local, explainable RAG index for codebases and coding agents
 Author: rag-your-code contributors
 License-Expression: MIT
@@ -76,14 +76,23 @@ package itself on first use.
 ```bash
 pip install rag-your-code
 
-rag-your-code index .
+rag-your-code bootstrap .                  # index, and say what is still missing
 rag-your-code search "where are HTTP retries handled" --json
 rag-your-code search "what calls the retry handler" --graph --hops 1 --json
 ```
 
+`bootstrap` exists because indexing a repository is not the same as making it
+searchable, and nothing used to say so. A fresh index retrieves against the
+sentence the parser generated, which adds no word the source did not already
+have. It reports which rung this repository is on — descriptions still to
+write, a promotion to apply, or nothing left — and hands over that rung's
+work. It reads the state rather than remembering a position, so running it
+again after each round is how you make progress. `index` still exists and does
+only the indexing.
+
 The index is written under `.rag-your-code/`; your source files are never
-modified. Later `index` runs reuse unchanged files. For a large repository,
-prefer `rag-your-code index . --compact`.
+modified. Later runs reuse unchanged files. For a large repository, prefer
+`--compact`.
 
 ## How it works
 
@@ -135,16 +144,45 @@ A trained embedding model scores row 2 at around 0.8. Here a synonym pair and
 an unrelated pair are indistinguishable, because no shared word is no shared
 word either way.
 
-Retrieval works regardless, because **identifiers and docstrings are already
-natural language** — `retry_charge` contains *retry* and *charge*. But it
-reaches only concepts somebody wrote down. Two things close the rest of the
-gap, and neither is a model:
+Retrieval works regardless, because **the prose people write about code is
+already natural language** — docstrings, comments, descriptions. Identifiers
+are not part of that, and it is worth being exact: `retry_charge` tokenizes to
+one opaque term, not to *retry* and *charge*. Splitting identifiers was
+implemented and measured against all three rulers, with query and stored
+vectors rebuilt together, and it was equal or worse on every one; the pieces it
+makes are `get`, `find`, `check`, `test`, which rarity weighting immediately
+discounts to nothing.
+
+So retrieval reaches only concepts somebody wrote down. Two things close the
+rest of the gap, and neither is a model:
 
 - **Your agent rewrites the query.** It has the conversation; turning
   "重试扣款" into `retry charge payment gateway` costs it nothing.
 - **Your agent writes the descriptions**, which puts the missing vocabulary
   into the index once instead of into every query.
 
+### Seven attempts to make the vector half earn its place
+
+Because "just use a better embedding" is the obvious next thought, it was
+measured rather than argued about. Six schemes were implemented — character
+n-grams, corpus co-occurrence via random indexing, truncated SVD, posting-list
+signatures, a rarity- and field-weighted hash, and call-graph diffusion — plus
+lexical postings expansion and embedding only the authored text. **On the
+foreign-repository ruler, not one of them beat using no vector at all.**
+
+The reason is architectural, not representational. Retrieval scores only the
+units the lexical half already matched, so a vector can reorder an answer but
+can never make one *retrievable*; pure cosine fires only when nothing matched
+at all, on 1 question of 35. Every scheme was competing for the same one- or
+two-question reshuffle inside a list that had already been chosen.
+
+Two things follow, and both are stated here rather than buried. Corpus-learned
+semantics need orders of magnitude more text than a repository has: 65% of the
+foreign corpus's terms appear in four or fewer units, so their co-occurrence
+row is a handful of sightings rather than a distribution. And on a *described*
+repository the shipped hash is useful precisely because it is blunt — every
+scheme that sharpened it lost ground there.
+
 ## Agent-authored descriptions
 
 Every unit carries a description, and that description is indexed. By default
@@ -330,6 +368,7 @@ full rebuild. The rest take effect immediately and invalidate nothing.
 one reply per line:
 
 ```json
+{"action":"bootstrap"}
 {"action":"search","query":"database transaction rollback","limit":5}
 {"action":"research","query":"trace payment retry behavior","max_steps":2}
 {"action":"neighbors","id":"payments.py:4:retry_charge","hops":1}
@@ -340,6 +379,14 @@ one reply per line:
 {"action":"stats"}
 ```
 
+**A result is navigation, not the file.** `results` carries the identifier,
+path, line range, signature, description, score and matched terms. The code
+arrives once, in the reply's `context`, trimmed to `max_chars`, and
+`omitted_for_budget` says how many results it did not reach. Carrying the
+source per result as well is what let one `search --json` reply reach 65,025
+characters against a stated budget of 12,000, and one `research` reply reach
+111,843 by serialising the same eight units three times over.
+
 **No single request can end the session.** Numeric fields saturate at their
 bounds, `open` is bounded in both lines and bytes, and anything unanticipated
 is reported in-band with its exception type. Streams are pinned to UTF-8

{rag_your_code-0.6.0 → rag_your_code-0.7.0}/README.md

@@ -49,14 +49,23 @@ package itself on first use.
 ```bash
 pip install rag-your-code
 
-rag-your-code index .
+rag-your-code bootstrap .                  # index, and say what is still missing
 rag-your-code search "where are HTTP retries handled" --json
 rag-your-code search "what calls the retry handler" --graph --hops 1 --json
 ```
 
+`bootstrap` exists because indexing a repository is not the same as making it
+searchable, and nothing used to say so. A fresh index retrieves against the
+sentence the parser generated, which adds no word the source did not already
+have. It reports which rung this repository is on — descriptions still to
+write, a promotion to apply, or nothing left — and hands over that rung's
+work. It reads the state rather than remembering a position, so running it
+again after each round is how you make progress. `index` still exists and does
+only the indexing.
+
 The index is written under `.rag-your-code/`; your source files are never
-modified. Later `index` runs reuse unchanged files. For a large repository,
-prefer `rag-your-code index . --compact`.
+modified. Later runs reuse unchanged files. For a large repository, prefer
+`--compact`.
 
 ## How it works
 
@@ -108,16 +117,45 @@ A trained embedding model scores row 2 at around 0.8. Here a synonym pair and
 an unrelated pair are indistinguishable, because no shared word is no shared
 word either way.
 
-Retrieval works regardless, because **identifiers and docstrings are already
-natural language** — `retry_charge` contains *retry* and *charge*. But it
-reaches only concepts somebody wrote down. Two things close the rest of the
-gap, and neither is a model:
+Retrieval works regardless, because **the prose people write about code is
+already natural language** — docstrings, comments, descriptions. Identifiers
+are not part of that, and it is worth being exact: `retry_charge` tokenizes to
+one opaque term, not to *retry* and *charge*. Splitting identifiers was
+implemented and measured against all three rulers, with query and stored
+vectors rebuilt together, and it was equal or worse on every one; the pieces it
+makes are `get`, `find`, `check`, `test`, which rarity weighting immediately
+discounts to nothing.
+
+So retrieval reaches only concepts somebody wrote down. Two things close the
+rest of the gap, and neither is a model:
 
 - **Your agent rewrites the query.** It has the conversation; turning
   "重试扣款" into `retry charge payment gateway` costs it nothing.
 - **Your agent writes the descriptions**, which puts the missing vocabulary
   into the index once instead of into every query.
 
+### Seven attempts to make the vector half earn its place
+
+Because "just use a better embedding" is the obvious next thought, it was
+measured rather than argued about. Six schemes were implemented — character
+n-grams, corpus co-occurrence via random indexing, truncated SVD, posting-list
+signatures, a rarity- and field-weighted hash, and call-graph diffusion — plus
+lexical postings expansion and embedding only the authored text. **On the
+foreign-repository ruler, not one of them beat using no vector at all.**
+
+The reason is architectural, not representational. Retrieval scores only the
+units the lexical half already matched, so a vector can reorder an answer but
+can never make one *retrievable*; pure cosine fires only when nothing matched
+at all, on 1 question of 35. Every scheme was competing for the same one- or
+two-question reshuffle inside a list that had already been chosen.
+
+Two things follow, and both are stated here rather than buried. Corpus-learned
+semantics need orders of magnitude more text than a repository has: 65% of the
+foreign corpus's terms appear in four or fewer units, so their co-occurrence
+row is a handful of sightings rather than a distribution. And on a *described*
+repository the shipped hash is useful precisely because it is blunt — every
+scheme that sharpened it lost ground there.
+
 ## Agent-authored descriptions
 
 Every unit carries a description, and that description is indexed. By default
@@ -303,6 +341,7 @@ full rebuild. The rest take effect immediately and invalidate nothing.
 one reply per line:
 
 ```json
+{"action":"bootstrap"}
 {"action":"search","query":"database transaction rollback","limit":5}
 {"action":"research","query":"trace payment retry behavior","max_steps":2}
 {"action":"neighbors","id":"payments.py:4:retry_charge","hops":1}
@@ -313,6 +352,14 @@ one reply per line:
 {"action":"stats"}
 ```
 
+**A result is navigation, not the file.** `results` carries the identifier,
+path, line range, signature, description, score and matched terms. The code
+arrives once, in the reply's `context`, trimmed to `max_chars`, and
+`omitted_for_budget` says how many results it did not reach. Carrying the
+source per result as well is what let one `search --json` reply reach 65,025
+characters against a stated budget of 12,000, and one `research` reply reach
+111,843 by serialising the same eight units three times over.
+
 **No single request can end the session.** Numeric fields saturate at their
 bounds, `open` is bounded in both lines and bytes, and anything unanticipated
 is reported in-band with its exception type. Streams are pinned to UTF-8

{rag_your_code-0.6.0 → rag_your_code-0.7.0}/pyproject.toml

@@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "rag-your-code"
-version = "0.6.0"
+version = "0.7.0"
 description = "A local, explainable RAG index for codebases and coding agents"
 readme = "README.md"
 requires-python = ">=3.10"

{rag_your_code-0.6.0 → rag_your_code-0.7.0/src/rag_your_code.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rag-your-code
-Version: 0.6.0
+Version: 0.7.0
 Summary: A local, explainable RAG index for codebases and coding agents
 Author: rag-your-code contributors
 License-Expression: MIT
@@ -76,14 +76,23 @@ package itself on first use.
 ```bash
 pip install rag-your-code
 
-rag-your-code index .
+rag-your-code bootstrap .                  # index, and say what is still missing
 rag-your-code search "where are HTTP retries handled" --json
 rag-your-code search "what calls the retry handler" --graph --hops 1 --json
 ```
 
+`bootstrap` exists because indexing a repository is not the same as making it
+searchable, and nothing used to say so. A fresh index retrieves against the
+sentence the parser generated, which adds no word the source did not already
+have. It reports which rung this repository is on — descriptions still to
+write, a promotion to apply, or nothing left — and hands over that rung's
+work. It reads the state rather than remembering a position, so running it
+again after each round is how you make progress. `index` still exists and does
+only the indexing.
+
 The index is written under `.rag-your-code/`; your source files are never
-modified. Later `index` runs reuse unchanged files. For a large repository,
-prefer `rag-your-code index . --compact`.
+modified. Later runs reuse unchanged files. For a large repository, prefer
+`--compact`.
 
 ## How it works
 
@@ -135,16 +144,45 @@ A trained embedding model scores row 2 at around 0.8. Here a synonym pair and
 an unrelated pair are indistinguishable, because no shared word is no shared
 word either way.
 
-Retrieval works regardless, because **identifiers and docstrings are already
-natural language** — `retry_charge` contains *retry* and *charge*. But it
-reaches only concepts somebody wrote down. Two things close the rest of the
-gap, and neither is a model:
+Retrieval works regardless, because **the prose people write about code is
+already natural language** — docstrings, comments, descriptions. Identifiers
+are not part of that, and it is worth being exact: `retry_charge` tokenizes to
+one opaque term, not to *retry* and *charge*. Splitting identifiers was
+implemented and measured against all three rulers, with query and stored
+vectors rebuilt together, and it was equal or worse on every one; the pieces it
+makes are `get`, `find`, `check`, `test`, which rarity weighting immediately
+discounts to nothing.
+
+So retrieval reaches only concepts somebody wrote down. Two things close the
+rest of the gap, and neither is a model:
 
 - **Your agent rewrites the query.** It has the conversation; turning
   "重试扣款" into `retry charge payment gateway` costs it nothing.
 - **Your agent writes the descriptions**, which puts the missing vocabulary
   into the index once instead of into every query.
 
+### Seven attempts to make the vector half earn its place
+
+Because "just use a better embedding" is the obvious next thought, it was
+measured rather than argued about. Six schemes were implemented — character
+n-grams, corpus co-occurrence via random indexing, truncated SVD, posting-list
+signatures, a rarity- and field-weighted hash, and call-graph diffusion — plus
+lexical postings expansion and embedding only the authored text. **On the
+foreign-repository ruler, not one of them beat using no vector at all.**
+
+The reason is architectural, not representational. Retrieval scores only the
+units the lexical half already matched, so a vector can reorder an answer but
+can never make one *retrievable*; pure cosine fires only when nothing matched
+at all, on 1 question of 35. Every scheme was competing for the same one- or
+two-question reshuffle inside a list that had already been chosen.
+
+Two things follow, and both are stated here rather than buried. Corpus-learned
+semantics need orders of magnitude more text than a repository has: 65% of the
+foreign corpus's terms appear in four or fewer units, so their co-occurrence
+row is a handful of sightings rather than a distribution. And on a *described*
+repository the shipped hash is useful precisely because it is blunt — every
+scheme that sharpened it lost ground there.
+
 ## Agent-authored descriptions
 
 Every unit carries a description, and that description is indexed. By default
@@ -330,6 +368,7 @@ full rebuild. The rest take effect immediately and invalidate nothing.
 one reply per line:
 
 ```json
+{"action":"bootstrap"}
 {"action":"search","query":"database transaction rollback","limit":5}
 {"action":"research","query":"trace payment retry behavior","max_steps":2}
 {"action":"neighbors","id":"payments.py:4:retry_charge","hops":1}
@@ -340,6 +379,14 @@ one reply per line:
 {"action":"stats"}
 ```
 
+**A result is navigation, not the file.** `results` carries the identifier,
+path, line range, signature, description, score and matched terms. The code
+arrives once, in the reply's `context`, trimmed to `max_chars`, and
+`omitted_for_budget` says how many results it did not reach. Carrying the
+source per result as well is what let one `search --json` reply reach 65,025
+characters against a stated budget of 12,000, and one `research` reply reach
+111,843 by serialising the same eight units three times over.
+
 **No single request can end the session.** Numeric fields saturate at their
 bounds, `open` is bounded in both lines and bytes, and anything unanticipated
 is reported in-band with its exception type. Streams are pinned to UTF-8

{rag_your_code-0.6.0 → rag_your_code-0.7.0}/src/rag_your_code.egg-info/SOURCES.txt

@@ -21,6 +21,7 @@ src/ragyourcode/models.py
 src/ragyourcode/parser.py
 src/ragyourcode/py.typed
 src/ragyourcode/search.py
+src/ragyourcode/workflow.py
 tests/test_agent_protocol.py
 tests/test_agentic.py
 tests/test_config.py
@@ -39,4 +40,5 @@ tests/test_ragyourcode.py
 tests/test_ranking.py
 tests/test_repo_queries.py
 tests/test_resilience.py
-tests/test_retrieval_correctness.py
+tests/test_retrieval_correctness.py
+tests/test_workflow.py

{rag_your_code-0.6.0 → rag_your_code-0.7.0}/src/ragyourcode/__init__.py

@@ -3,4 +3,4 @@
 from .models import CodeUnit, SearchResult
 
 __all__ = ["CodeUnit", "SearchResult"]
-__version__ = "0.6.0"
+__version__ = "0.7.0"

rag_your_code-0.7.0/src/ragyourcode/agentic.py

@@ -0,0 +1,118 @@
+"""Bounded, observable agentic retrieval (ARAG) orchestration."""
+
+from __future__ import annotations
+
+from .graph import CodeGraph, graph_search
+from .models import CodeUnit, SearchResult
+from .search import DEFAULT_VECTOR_WEIGHT, SearchIndex, context, search, within_budget
+
+
+def _result_ids(results: list[SearchResult]) -> set[str]:
+    """Collects the unit identifiers out of a list of search results, so two
+    result sets can be compared for overlap or novelty. Used to decide
+    whether a second retrieval round actually surfaced anything new.
+    """
+    return {result.unit.id for result in results}
+
+
+# How far ahead of the runner-up the top result must be before a second
+# retrieval step is skipped, as a fraction of the top score.
+#
+# This replaced an absolute `confidence_threshold` of 0.8, which was a number
+# tied to a scoring scale. When ranking became BM25F the scale moved and the
+# threshold quietly died: measured across 105 ruler questions, only 3% of
+# queries reached 0.8 at all, so the early stop had stopped existing and every
+# research call ran the graph expansion. A margin is a ratio between two scores
+# from the same query, so it cannot drift when the scoring changes again.
+#
+# Measured over those 105 questions: top-1 is correct 42% of the time overall,
+# and 68-73% of the time among the queries this fires on. Anywhere in 0.2-0.5
+# behaves the same at this sample size; 0.30 is the middle of that band, not a
+# measured optimum.
+DEFAULT_DOMINANCE: float = 0.30
+
+
+def _serialize(results: list[SearchResult]) -> list[dict]:
+    """Converts search results into plain JSON-ready dictionaries for the agent
+    protocol reply.
+    """
+    return [result.to_dict() for result in results]
+
+
+def _trace(results: list[SearchResult]) -> list[dict]:
+    """One step of the search, as a trace rather than as a payload.
+
+    A step exists so the caller can see how the answer was reached: which
+    units each round reached and how strongly. That needs an identifier, a
+    score and the matching words -- not a second and third copy of every
+    unit's description, callees and imports. Reporting two steps in full is
+    what made a research reply three times the size of the answer inside it.
+    """
+    return [
+        {"id": result.unit.id, "score": round(result.score, 6), "matched_terms": result.matched_terms}
+        for result in results
+    ]
+
+
+def dominance(results: list[SearchResult]) -> float:
+    """How far the top result is ahead of the runner-up, relative to the top.
+
+    One result alone is unopposed and scores 1.0. Nothing scores 0.0. The
+    quantity is scale-free by construction, which is the whole point: it
+    compares two numbers produced by the same query under the same scoring
+    rule, so no future change to that rule can silently recalibrate it.
+    """
+    if not results:
+        return 0.0
+    top = results[0].score
+    if len(results) == 1:
+        return 1.0
+    return (top - results[1].score) / top if top else 0.0
+
+
+def research(
+    units: list[CodeUnit],
+    query: str,
+    limit: int = 8,
+    hops: int = 1,
+    max_steps: int = 2,
+    dominance_threshold: float = DEFAULT_DOMINANCE,
+    graph: CodeGraph | None = None,
+    search_index: SearchIndex | None = None,
+    vector_weight: float = DEFAULT_VECTOR_WEIGHT,
+    max_chars: int = 12000,
+) -> dict:
+    """Run at most two deterministic retrieval steps and explain the stop.
+
+    This is deliberately bounded. A future LLM planner can replace the query
+    proposal, but the budget, evidence format, and no-progress stop remain
+    stable safety contracts.
+
+    The reply carries the code once, in a context block trimmed to
+    ``max_chars``. Results are navigation only. Reporting two retrieval steps
+    used to mean serialising the same eight units three times with their
+    source attached, which is how one research answer reached 111,843
+    characters against a stated budget of 12,000.
+    """
+    max_steps = min(2, max(1, max_steps))
+    steps: list[dict] = []
+    initial = search(units, query, max(limit, 1), search_index=search_index, vector_weight=vector_weight)
+    steps.append({"action": "search", "query": query, "results": _trace(initial)})
+    if not initial:
+        return {"query": query, "results": [], "steps": steps, "stop_reason": "no_results", "context": ""}
+    unopposed = dominance(initial) >= dominance_threshold and bool(initial[0].matched_terms)
+    if max_steps == 1 or unopposed:
+        kept = initial[:limit]
+        return {"query": query, "results": _serialize(kept), "steps": steps, "stop_reason": "high_confidence", "context": context(within_budget(kept, max_chars), max_chars)}
+
+    expanded = graph_search(units, query, limit=max(limit * 2, 8), hops=hops, graph=graph, search_index=search_index, vector_weight=vector_weight)
+    steps.append({"action": "graph_expand", "hops": hops, "results": _trace(expanded[:limit])})
+    merged = {result.unit.id: result for result in initial}
+    for result in expanded:
+        current = merged.get(result.unit.id)
+        if current is None or result.score > current.score:
+            merged[result.unit.id] = result
+    final = sorted(merged.values(), key=lambda result: (-result.score, result.unit.id))[:limit]
+    new_ids = _result_ids(final) - _result_ids(initial)
+    reason = "new_graph_evidence" if new_ids else "no_new_evidence"
+    return {"query": query, "results": _serialize(final), "steps": steps, "stop_reason": reason, "context": context(within_budget(final, max_chars), max_chars)}