rag-your-code 0.4.2__tar.gz → 0.6.0__tar.gz

{rag_your_code-0.4.2/src/rag_your_code.egg-info → rag_your_code-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rag-your-code
-Version: 0.4.2
+Version: 0.6.0
 Summary: A local, explainable RAG index for codebases and coding agents
 Author: rag-your-code contributors
 License-Expression: MIT
@@ -153,7 +153,15 @@ callees listed, the docstring appended. It introduces no vocabulary the source
 did not already have — which is exactly why retrieval cannot reach a concept
 nobody wrote down.
 
-The agent already reading the index can supply those words:
+**First, the documentation you already wrote is indexed.** Fourteen of the
+fifteen supported languages put documentation immediately above a declaration
+— JSDoc, Javadoc, KDoc, rustdoc, Go doc comments, XML doc comments, PHPDoc —
+and a unit's span begins at the declaration, so all of it used to sit outside
+the index. The same sentence reached thirteen searchable words as a Python
+docstring and two as a JavaScript comment. Now both reach thirteen. Commented-
+out code, separator rules and licence headers are deliberately left out.
+
+**Where there is none, the agent can write it:**
 
 ```bash
 rag-your-code describe status              # coverage, and what is pending
@@ -165,23 +173,73 @@ rag-your-code index .                      # apply it
 or, in the protocol, `describe_pending` and `describe_put` — which take effect
 in the same session, with no refresh.
 
+**And you can move it into the code**, where it needs no bookkeeping at all:
+
+```bash
+rag-your-code describe promote | git apply    # review it first
+```
+
+That emits a unified diff adding a doc comment in each language's own
+convention, for declarations that have none. The tool still never writes your
+source. Only the half meant for a reader is promoted, so a bilingual
+description leaves its second language in the store where retrieval still uses
+it — measured, promoting all 68 on this repository discarded no description
+and left Chinese retrieval unchanged.
+
 ### Measured on this repository
 
-This project describes its own implementation: all 120 units under `src/`
-carry an agent-written bilingual description, committed to the repo. Ten
-natural-language questions about the codebase, before and after:
+This project describes its own implementation: every unit under `src/` carries
+an agent-written bilingual description, committed to the repo, and 68 of them
+have been promoted into the source as doc comments.
 
-| | before | after |
-|---|---|---|
-| top result in the expected file | 2 / 8 | **6 / 8** |
-| Chinese-language queries | 0 / 4 | **3 / 4** |
-| queries with **no shared word at all** (pure fallback) | 4 | **0** |
+Seventy natural-language questions about this codebase, in English and
+Chinese, each listing every unit that genuinely answers it
+([`benchmarks/repo_queries.json`](benchmarks/repo_queries.json)):
 
-Two still miss, and both are worth stating. One query says `catastrophic
-backtracking` where the description says `backtracks catastrophically`: there
-is no stemming, so those share no word — exactly the limit documented above.
-The other returned a unit that answers the question from a different file than
-predicted, so the expectation was wrong rather than the retrieval.
+| | generated descriptions | agent-written |
+|---|---|---|
+| hit@1 | 0.271 | **0.500** |
+| hit@3 | 0.486 | **0.800** |
+| MRR | 0.367 | **0.631** |
+| answered with no shared word at all | 12.9% | **0%** |
+
+Roughly double the first-place accuracy. Fourteen questions still fail, which
+is what makes the set usable for measuring the next change;
+`tests/test_repo_queries.py` asserts that some question always does, and that
+the written column beats the generated one.
+
+One failure is worth naming: a query saying `catastrophic backtracking` does
+not reach a description saying `backtracks catastrophically`. There is no
+stemming — exactly the limit documented above.
+
+### Measured on a repository nobody here wrote
+
+The table above is the warmest case this project supports: its own code, its
+own descriptions, and questions written by the same party. It cannot say what
+a first-time user gets. So there is a second ruler — thirty-five questions
+about [cc-enforcer](https://github.com/skymanbp/cc-enforcer), 1153 units, no
+descriptions at all, each question phrased in a user's words rather than in
+the words of the docstring that answers it
+([`benchmarks/cold_queries.json`](benchmarks/cold_queries.json)):
+
+| | before 0.6.0 | now |
+|---|---|---|
+| hit@1 | 0.086 | **0.257** |
+| hit@3 | 0.229 | **0.400** |
+| MRR | 0.157 | **0.314** |
+
+Three times the first-place accuracy, and the same change moved both other
+rulers in the same direction. What it fixed was ranking: scoring used to be
+the fraction of query words a unit contained, so `the` counted for as much as
+`daemon`, and nothing corrected for size — the single largest declaration in
+that repository came back in the top three for four questions out of six. It
+is now BM25 over weighted fields, where a word's worth comes from how rare it
+is in *your* corpus and a word in a declaration's name outweighs the same word
+buried in a body.
+
+Twenty-one of the thirty-five still fail, and the largest remaining cause is
+named in [docs/TESTING.md](docs/TESTING.md): a test declaration often outranks
+the code it tests, because it repeats that code's vocabulary and adds its own.
 
 **What this is:** it moves the semantic work from query time to index time.
 Matching stays lexical. It is LLM-authored keyword expansion, and its reach is
@@ -230,7 +288,7 @@ Directional local measurements, not service levels; the archived run is
 **Suite:** Python 3.10 – 3.13 on Linux and Windows, plus a job that installs
 the built wheel into a clean environment and runs every command the
 documentation prescribes, and another that runs the skill's own install line
-verbatim. 208 tests as of 0.4.2 — the count is version-stamped rather than
+verbatim. 248 tests as of 0.5.0 — the count is version-stamped rather than
 maintained, because a bare figure in a living document is a claim that rots;
 per-release counts are in [CHANGELOG.md](CHANGELOG.md).
 

{rag_your_code-0.4.2 → rag_your_code-0.6.0}/README.md

@@ -126,7 +126,15 @@ callees listed, the docstring appended. It introduces no vocabulary the source
 did not already have — which is exactly why retrieval cannot reach a concept
 nobody wrote down.
 
-The agent already reading the index can supply those words:
+**First, the documentation you already wrote is indexed.** Fourteen of the
+fifteen supported languages put documentation immediately above a declaration
+— JSDoc, Javadoc, KDoc, rustdoc, Go doc comments, XML doc comments, PHPDoc —
+and a unit's span begins at the declaration, so all of it used to sit outside
+the index. The same sentence reached thirteen searchable words as a Python
+docstring and two as a JavaScript comment. Now both reach thirteen. Commented-
+out code, separator rules and licence headers are deliberately left out.
+
+**Where there is none, the agent can write it:**
 
 ```bash
 rag-your-code describe status              # coverage, and what is pending
@@ -138,23 +146,73 @@ rag-your-code index .                      # apply it
 or, in the protocol, `describe_pending` and `describe_put` — which take effect
 in the same session, with no refresh.
 
+**And you can move it into the code**, where it needs no bookkeeping at all:
+
+```bash
+rag-your-code describe promote | git apply    # review it first
+```
+
+That emits a unified diff adding a doc comment in each language's own
+convention, for declarations that have none. The tool still never writes your
+source. Only the half meant for a reader is promoted, so a bilingual
+description leaves its second language in the store where retrieval still uses
+it — measured, promoting all 68 on this repository discarded no description
+and left Chinese retrieval unchanged.
+
 ### Measured on this repository
 
-This project describes its own implementation: all 120 units under `src/`
-carry an agent-written bilingual description, committed to the repo. Ten
-natural-language questions about the codebase, before and after:
+This project describes its own implementation: every unit under `src/` carries
+an agent-written bilingual description, committed to the repo, and 68 of them
+have been promoted into the source as doc comments.
 
-| | before | after |
-|---|---|---|
-| top result in the expected file | 2 / 8 | **6 / 8** |
-| Chinese-language queries | 0 / 4 | **3 / 4** |
-| queries with **no shared word at all** (pure fallback) | 4 | **0** |
+Seventy natural-language questions about this codebase, in English and
+Chinese, each listing every unit that genuinely answers it
+([`benchmarks/repo_queries.json`](benchmarks/repo_queries.json)):
 
-Two still miss, and both are worth stating. One query says `catastrophic
-backtracking` where the description says `backtracks catastrophically`: there
-is no stemming, so those share no word — exactly the limit documented above.
-The other returned a unit that answers the question from a different file than
-predicted, so the expectation was wrong rather than the retrieval.
+| | generated descriptions | agent-written |
+|---|---|---|
+| hit@1 | 0.271 | **0.500** |
+| hit@3 | 0.486 | **0.800** |
+| MRR | 0.367 | **0.631** |
+| answered with no shared word at all | 12.9% | **0%** |
+
+Roughly double the first-place accuracy. Fourteen questions still fail, which
+is what makes the set usable for measuring the next change;
+`tests/test_repo_queries.py` asserts that some question always does, and that
+the written column beats the generated one.
+
+One failure is worth naming: a query saying `catastrophic backtracking` does
+not reach a description saying `backtracks catastrophically`. There is no
+stemming — exactly the limit documented above.
+
+### Measured on a repository nobody here wrote
+
+The table above is the warmest case this project supports: its own code, its
+own descriptions, and questions written by the same party. It cannot say what
+a first-time user gets. So there is a second ruler — thirty-five questions
+about [cc-enforcer](https://github.com/skymanbp/cc-enforcer), 1153 units, no
+descriptions at all, each question phrased in a user's words rather than in
+the words of the docstring that answers it
+([`benchmarks/cold_queries.json`](benchmarks/cold_queries.json)):
+
+| | before 0.6.0 | now |
+|---|---|---|
+| hit@1 | 0.086 | **0.257** |
+| hit@3 | 0.229 | **0.400** |
+| MRR | 0.157 | **0.314** |
+
+Three times the first-place accuracy, and the same change moved both other
+rulers in the same direction. What it fixed was ranking: scoring used to be
+the fraction of query words a unit contained, so `the` counted for as much as
+`daemon`, and nothing corrected for size — the single largest declaration in
+that repository came back in the top three for four questions out of six. It
+is now BM25 over weighted fields, where a word's worth comes from how rare it
+is in *your* corpus and a word in a declaration's name outweighs the same word
+buried in a body.
+
+Twenty-one of the thirty-five still fail, and the largest remaining cause is
+named in [docs/TESTING.md](docs/TESTING.md): a test declaration often outranks
+the code it tests, because it repeats that code's vocabulary and adds its own.
 
 **What this is:** it moves the semantic work from query time to index time.
 Matching stays lexical. It is LLM-authored keyword expansion, and its reach is
@@ -203,7 +261,7 @@ Directional local measurements, not service levels; the archived run is
 **Suite:** Python 3.10 – 3.13 on Linux and Windows, plus a job that installs
 the built wheel into a clean environment and runs every command the
 documentation prescribes, and another that runs the skill's own install line
-verbatim. 208 tests as of 0.4.2 — the count is version-stamped rather than
+verbatim. 248 tests as of 0.5.0 — the count is version-stamped rather than
 maintained, because a bare figure in a living document is a claim that rots;
 per-release counts are in [CHANGELOG.md](CHANGELOG.md).
 

{rag_your_code-0.4.2 → rag_your_code-0.6.0}/pyproject.toml

@@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "rag-your-code"
-version = "0.4.2"
+version = "0.6.0"
 description = "A local, explainable RAG index for codebases and coding agents"
 readme = "README.md"
 requires-python = ">=3.10"

{rag_your_code-0.4.2 → rag_your_code-0.6.0/src/rag_your_code.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rag-your-code
-Version: 0.4.2
+Version: 0.6.0
 Summary: A local, explainable RAG index for codebases and coding agents
 Author: rag-your-code contributors
 License-Expression: MIT
@@ -153,7 +153,15 @@ callees listed, the docstring appended. It introduces no vocabulary the source
 did not already have — which is exactly why retrieval cannot reach a concept
 nobody wrote down.
 
-The agent already reading the index can supply those words:
+**First, the documentation you already wrote is indexed.** Fourteen of the
+fifteen supported languages put documentation immediately above a declaration
+— JSDoc, Javadoc, KDoc, rustdoc, Go doc comments, XML doc comments, PHPDoc —
+and a unit's span begins at the declaration, so all of it used to sit outside
+the index. The same sentence reached thirteen searchable words as a Python
+docstring and two as a JavaScript comment. Now both reach thirteen. Commented-
+out code, separator rules and licence headers are deliberately left out.
+
+**Where there is none, the agent can write it:**
 
 ```bash
 rag-your-code describe status              # coverage, and what is pending
@@ -165,23 +173,73 @@ rag-your-code index .                      # apply it
 or, in the protocol, `describe_pending` and `describe_put` — which take effect
 in the same session, with no refresh.
 
+**And you can move it into the code**, where it needs no bookkeeping at all:
+
+```bash
+rag-your-code describe promote | git apply    # review it first
+```
+
+That emits a unified diff adding a doc comment in each language's own
+convention, for declarations that have none. The tool still never writes your
+source. Only the half meant for a reader is promoted, so a bilingual
+description leaves its second language in the store where retrieval still uses
+it — measured, promoting all 68 on this repository discarded no description
+and left Chinese retrieval unchanged.
+
 ### Measured on this repository
 
-This project describes its own implementation: all 120 units under `src/`
-carry an agent-written bilingual description, committed to the repo. Ten
-natural-language questions about the codebase, before and after:
+This project describes its own implementation: every unit under `src/` carries
+an agent-written bilingual description, committed to the repo, and 68 of them
+have been promoted into the source as doc comments.
 
-| | before | after |
-|---|---|---|
-| top result in the expected file | 2 / 8 | **6 / 8** |
-| Chinese-language queries | 0 / 4 | **3 / 4** |
-| queries with **no shared word at all** (pure fallback) | 4 | **0** |
+Seventy natural-language questions about this codebase, in English and
+Chinese, each listing every unit that genuinely answers it
+([`benchmarks/repo_queries.json`](benchmarks/repo_queries.json)):
 
-Two still miss, and both are worth stating. One query says `catastrophic
-backtracking` where the description says `backtracks catastrophically`: there
-is no stemming, so those share no word — exactly the limit documented above.
-The other returned a unit that answers the question from a different file than
-predicted, so the expectation was wrong rather than the retrieval.
+| | generated descriptions | agent-written |
+|---|---|---|
+| hit@1 | 0.271 | **0.500** |
+| hit@3 | 0.486 | **0.800** |
+| MRR | 0.367 | **0.631** |
+| answered with no shared word at all | 12.9% | **0%** |
+
+Roughly double the first-place accuracy. Fourteen questions still fail, which
+is what makes the set usable for measuring the next change;
+`tests/test_repo_queries.py` asserts that some question always does, and that
+the written column beats the generated one.
+
+One failure is worth naming: a query saying `catastrophic backtracking` does
+not reach a description saying `backtracks catastrophically`. There is no
+stemming — exactly the limit documented above.
+
+### Measured on a repository nobody here wrote
+
+The table above is the warmest case this project supports: its own code, its
+own descriptions, and questions written by the same party. It cannot say what
+a first-time user gets. So there is a second ruler — thirty-five questions
+about [cc-enforcer](https://github.com/skymanbp/cc-enforcer), 1153 units, no
+descriptions at all, each question phrased in a user's words rather than in
+the words of the docstring that answers it
+([`benchmarks/cold_queries.json`](benchmarks/cold_queries.json)):
+
+| | before 0.6.0 | now |
+|---|---|---|
+| hit@1 | 0.086 | **0.257** |
+| hit@3 | 0.229 | **0.400** |
+| MRR | 0.157 | **0.314** |
+
+Three times the first-place accuracy, and the same change moved both other
+rulers in the same direction. What it fixed was ranking: scoring used to be
+the fraction of query words a unit contained, so `the` counted for as much as
+`daemon`, and nothing corrected for size — the single largest declaration in
+that repository came back in the top three for four questions out of six. It
+is now BM25 over weighted fields, where a word's worth comes from how rare it
+is in *your* corpus and a word in a declaration's name outweighs the same word
+buried in a body.
+
+Twenty-one of the thirty-five still fail, and the largest remaining cause is
+named in [docs/TESTING.md](docs/TESTING.md): a test declaration often outranks
+the code it tests, because it repeats that code's vocabulary and adds its own.
 
 **What this is:** it moves the semantic work from query time to index time.
 Matching stays lexical. It is LLM-authored keyword expansion, and its reach is
@@ -230,7 +288,7 @@ Directional local measurements, not service levels; the archived run is
 **Suite:** Python 3.10 – 3.13 on Linux and Windows, plus a job that installs
 the built wheel into a clean environment and runs every command the
 documentation prescribes, and another that runs the skill's own install line
-verbatim. 208 tests as of 0.4.2 — the count is version-stamped rather than
+verbatim. 248 tests as of 0.5.0 — the count is version-stamped rather than
 maintained, because a bare figure in a living document is a claim that rots;
 per-release counts are in [CHANGELOG.md](CHANGELOG.md).
 

{rag_your_code-0.4.2 → rag_your_code-0.6.0}/src/rag_your_code.egg-info/SOURCES.txt

@@ -13,6 +13,7 @@ src/ragyourcode/annotate.py
 src/ragyourcode/cli.py
 src/ragyourcode/config.py
 src/ragyourcode/descriptions.py
+src/ragyourcode/document.py
 src/ragyourcode/embeddings.py
 src/ragyourcode/graph.py
 src/ragyourcode/indexer.py
@@ -24,6 +25,8 @@ tests/test_agent_protocol.py
 tests/test_agentic.py
 tests/test_config.py
 tests/test_descriptions.py
+tests/test_doc_comments.py
+tests/test_document.py
 tests/test_e2e_cli.py
 tests/test_golden.py
 tests/test_graph_incremental.py
@@ -33,5 +36,7 @@ tests/test_metadata.py
 tests/test_multilanguage.py
 tests/test_parser_edges.py
 tests/test_ragyourcode.py
+tests/test_ranking.py
+tests/test_repo_queries.py
 tests/test_resilience.py
 tests/test_retrieval_correctness.py

{rag_your_code-0.4.2 → rag_your_code-0.6.0}/src/ragyourcode/__init__.py

@@ -3,4 +3,4 @@
 from .models import CodeUnit, SearchResult
 
 __all__ = ["CodeUnit", "SearchResult"]
-__version__ = "0.4.2"
+__version__ = "0.6.0"

{rag_your_code-0.4.2 → rag_your_code-0.6.0}/src/ragyourcode/agentic.py

@@ -8,10 +8,17 @@ from .search import DEFAULT_VECTOR_WEIGHT, SearchIndex, search
 
 
 def _result_ids(results: list[SearchResult]) -> set[str]:
+    """Collects the unit identifiers out of a list of search results, so two
+    result sets can be compared for overlap or novelty. Used to decide
+    whether a second retrieval round actually surfaced anything new.
+    """
     return {result.unit.id for result in results}
 
 
 def _serialize(results: list[SearchResult]) -> list[dict]:
+    """Converts search results into plain JSON-ready dictionaries for the agent
+    protocol reply.
+    """
     return [result.to_dict() for result in results]
 
 

{rag_your_code-0.4.2 → rag_your_code-0.6.0}/src/ragyourcode/annotate.py

@@ -7,11 +7,22 @@ import re
 
 
 def _humanize(name: str) -> str:
+    """Turns a programmer identifier into ordinary words: splits camelCase
+    apart, replaces underscores with spaces, lowercases the result. Empty
+    input becomes a placeholder rather than an empty string.
+    """
     words = re.sub(r"([a-z0-9])([A-Z])", r"\1 \2", name).replace("_", " ").split()
     return " ".join(words).strip().lower() or "anonymous unit"
 
 
 def describe_python(node: ast.AST, source: str, calls: list[str], imports: list[str]) -> str:
+    """Builds a readable sentence about a Python function or class without
+    using a language model: the humanised name, the arguments it accepts,
+    the functions it calls, the modules it uses, and the docstring appended
+    verbatim as stated intent. Because it only rearranges words already in
+    the source, it adds no vocabulary the code did not have, which is why
+    retrieval cannot reach a concept nobody wrote down.
+    """
     name = getattr(node, "name", "anonymous")
     kind = "method" if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) else "class"
     args = []