rag-your-code 1.4.0__tar.gz → 1.4.1__tar.gz

{rag_your_code-1.4.0/src/rag_your_code.egg-info → rag_your_code-1.4.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rag-your-code
-Version: 1.4.0
+Version: 1.4.1
 Summary: A local, explainable RAG index for codebases and coding agents
 Author: rag-your-code contributors
 License-Expression: MIT
@@ -295,11 +295,11 @@ repository had grown by ninety units.
 |---|---|---|---|---|---|
 | **A** Flask 3.1.3, no descriptions | what a first-time user gets | 35 | 0.200 | 0.286 | 0.238 |
 | **B** this repo, generated descriptions only | a cold index of familiar code | 70 | 0.314 | 0.471 | 0.383 |
-| **C** this repo, agent-written descriptions | the warmest case supported | 70 | 0.443 | 0.614 | 0.507 |
+| **C** this repo, agent-written descriptions | the warmest case supported | 70 | 0.443 | 0.614 | 0.509 |
 
 The corpora, without which none of the above is reproducible — **A** 1,572
-units, `5fd51169eacc`; **B** 581 units, `8e1e71942c1c`; **C** 581 units,
-`978a1d48a82a`. Ruler A grades **a copy of Flask 3.1.3 carried in this
+units, `5fd51169eacc`; **B** 584 units, `fb1f841fa43a`; **C** 584 units,
+`c9df00350cbd`. Ruler A grades **a copy of Flask 3.1.3 carried in this
 repository**, at [`benchmarks/corpus/flask`](benchmarks/corpus/flask), pinned
 to commit `22d9247`. Through 1.3.0 it graded a checkout on one machine, and
 that cost three things: two questions pointed at a declaration the subject had
@@ -330,10 +330,10 @@ Flask does not implement get through on exactly that.
 
 | gate | A hit@1/3/MRR | B hit@1/3/MRR | C hit@1/3/MRR | silence own / foreign |
 |---|---|---|---|---|
-| neither (pre-1.0.0) | 0.200/0.286/0.238 | 0.314/0.486/0.391 | 0.486/0.700/0.567 | 0.000 / 0.000 |
-| coverage only (1.0.0) | 0.200/0.286/0.238 | 0.314/0.471/0.383 | 0.486/0.686/0.562 | 0.600 / 0.733 |
-| concentration only | 0.200/0.286/0.238 | 0.314/0.471/0.383 | 0.443/0.614/0.507 | 0.967 / 0.800 |
-| **both (1.1.0)** | **0.200/0.286/0.238** | **0.314/0.471/0.383** | 0.443/0.614/0.507 | **0.967 / 0.833** |
+| neither (pre-1.0.0) | 0.200/0.286/0.238 | 0.314/0.486/0.391 | 0.486/0.700/0.569 | 0.000 / 0.000 |
+| coverage only (1.0.0) | 0.200/0.286/0.238 | 0.314/0.471/0.383 | 0.486/0.686/0.564 | 0.567 / 0.733 |
+| concentration only | 0.200/0.286/0.238 | 0.314/0.471/0.383 | 0.443/0.614/0.509 | 0.967 / 0.800 |
+| **both (1.1.0)** | **0.200/0.286/0.238** | **0.314/0.471/0.383** | 0.443/0.614/0.509 | **0.967 / 0.833** |
 
 Ruler A is **unmoved by either bar**, and B by concentration. The whole cost is
 three questions of seventy at hit@1 on the warmest ruler, and six at hit@3.
@@ -352,20 +352,20 @@ from 0.314 and C to 0.329 from 0.443. **0.28 was chosen before this corpus
 existed and survived meeting it**, which is the only kind of evidence a default
 can have.
 
-**Latency** — warm corpus, 581 units `978a1d48a82a`, five consecutive
+**Latency** — warm corpus, 584 units `c9df00350cbd`, five consecutive
 invocations of `python -m benchmarks.query_latency --repeats 10` (420 samples
 each):
 
 | | | across the five |
 |---|---|---|
-| query, median | **1.0 ms** | 0.71 – 1.26 |
-| query, p95 | 1.7 ms | 1.2 – 3.7 |
-| refusing an unanswerable query | **0.029 ms** | 0.023 – 0.031 |
-| refusal cheaper than answering by | **~36×** | 31 – 44 |
+| query, median | **0.61 ms** | 0.60 – 0.64 |
+| query, p95 | 1.09 ms | 1.06 – 1.14 |
+| refusing an unanswerable query | **0.017 ms** | 0.015 – 0.019 |
+| refusal cheaper than answering by | **~36×** | 33 – 41 |
 
 Two significant figures and a spread, because that is the precision the
-measurement has. Across fifteen invocations over two releases on the same idle
-machine the median has landed anywhere from 0.62 to 1.44 ms and p95 from 1.09
+measurement has. Across twenty invocations over three releases on the same idle
+machine the median has landed anywhere from 0.51 to 1.44 ms and p95 from 0.85
 to 7.34 ms — a band wider than any change the code has ever made to this
 number. Releases before 1.3.0 published `0.83 ms / p95 1.68 ms` to three
 figures from a script that was never committed; both values sit inside that
@@ -420,17 +420,17 @@ framework reversed it. The honest claim is narrower than either table alone:
 | right file first | 22.9% | **37.1%** |
 | right file in top 3 | 45.7% | **57.1%** |
 | lines it hands back, all questions | 17,641 | — |
-| characters returned, all questions | 1,415,656 | **249,720** |
+| characters returned, all questions | 1,415,656 | **258,236** |
 | questions it answers | 30 | 30 |
 
 **Once the vocabulary exists, it is not close.**
 
-| this repository · 70 questions · 581 units `978a1d48a82a` · 304 described | Grep loop | rag-your-code |
+| this repository · 70 questions · 584 units `c9df00350cbd` · 304 described | Grep loop | rag-your-code |
 |---|---|---|
 | right file first | 22.9% | **58.6%** |
-| right file in top 3 | 54.3% | **75.7%** |
-| lines it hands back, all questions | 11,833 | — |
-| characters returned, all questions | 1,122,902 | **626,022** |
+| right file in top 3 | 54.3% | **77.1%** |
+| lines it hands back, all questions | 11,959 | — |
+| characters returned, all questions | 1,135,411 | **615,673** |
 | questions it answers | **61** | 60 |
 
 Those two tables are the whole argument of section 3.3, measured against a real
@@ -467,11 +467,17 @@ Four qualifications, because the table would otherwise flatter both sides:
   `the` gets every file back in no order. It is also why Grep declines nine of
   the seventy questions here — those had no word left that this corpus does not
   use everywhere.
-- **Payload is counted in characters on both sides.** On this repository Grep
-  hands back 18,400 characters per question it answers, unranked and without
-  spans, against 10,400 here, ranked and capped by `search.max_chars` — a
-  factor of 1.8. On Flask it is a factor of 5.7 — 47,200 characters against
-  8,300 — because a framework repeats its own vocabulary across many files and
+- **Payload is counted in characters on both sides**, and 1.4.1 changed what
+  fits in it. A generated description ends with the author's docstring so the
+  docstring is searchable, and the source printed below it said the same thing
+  again — 2,381 of 3,382 characters of prose header on Flask were a verbatim
+  repeat of the code beneath. The block no longer prints what the code shows,
+  and the same 12,000-character budget now carries **119 declarations instead
+  of 92** there, 323 instead of 305 here. Grep
+  hands back 18,600 characters per question it answers, unranked and without
+  spans, against 10,300 here, ranked and capped by `search.max_chars` — a
+  factor of 1.8. On Flask it is a factor of 5.5 — 47,200 characters against
+  8,600 — because a framework repeats its own vocabulary across many files and
   Grep has no way to rank what it finds. Both sides decline the same five of
   those 35 — and they are the five Chinese ones, all of them. A Chinese word is
   not a substring of English source and it is not a token in an index built

{rag_your_code-1.4.0 → rag_your_code-1.4.1}/README.md

@@ -266,11 +266,11 @@ repository had grown by ninety units.
 |---|---|---|---|---|---|
 | **A** Flask 3.1.3, no descriptions | what a first-time user gets | 35 | 0.200 | 0.286 | 0.238 |
 | **B** this repo, generated descriptions only | a cold index of familiar code | 70 | 0.314 | 0.471 | 0.383 |
-| **C** this repo, agent-written descriptions | the warmest case supported | 70 | 0.443 | 0.614 | 0.507 |
+| **C** this repo, agent-written descriptions | the warmest case supported | 70 | 0.443 | 0.614 | 0.509 |
 
 The corpora, without which none of the above is reproducible — **A** 1,572
-units, `5fd51169eacc`; **B** 581 units, `8e1e71942c1c`; **C** 581 units,
-`978a1d48a82a`. Ruler A grades **a copy of Flask 3.1.3 carried in this
+units, `5fd51169eacc`; **B** 584 units, `fb1f841fa43a`; **C** 584 units,
+`c9df00350cbd`. Ruler A grades **a copy of Flask 3.1.3 carried in this
 repository**, at [`benchmarks/corpus/flask`](benchmarks/corpus/flask), pinned
 to commit `22d9247`. Through 1.3.0 it graded a checkout on one machine, and
 that cost three things: two questions pointed at a declaration the subject had
@@ -301,10 +301,10 @@ Flask does not implement get through on exactly that.
 
 | gate | A hit@1/3/MRR | B hit@1/3/MRR | C hit@1/3/MRR | silence own / foreign |
 |---|---|---|---|---|
-| neither (pre-1.0.0) | 0.200/0.286/0.238 | 0.314/0.486/0.391 | 0.486/0.700/0.567 | 0.000 / 0.000 |
-| coverage only (1.0.0) | 0.200/0.286/0.238 | 0.314/0.471/0.383 | 0.486/0.686/0.562 | 0.600 / 0.733 |
-| concentration only | 0.200/0.286/0.238 | 0.314/0.471/0.383 | 0.443/0.614/0.507 | 0.967 / 0.800 |
-| **both (1.1.0)** | **0.200/0.286/0.238** | **0.314/0.471/0.383** | 0.443/0.614/0.507 | **0.967 / 0.833** |
+| neither (pre-1.0.0) | 0.200/0.286/0.238 | 0.314/0.486/0.391 | 0.486/0.700/0.569 | 0.000 / 0.000 |
+| coverage only (1.0.0) | 0.200/0.286/0.238 | 0.314/0.471/0.383 | 0.486/0.686/0.564 | 0.567 / 0.733 |
+| concentration only | 0.200/0.286/0.238 | 0.314/0.471/0.383 | 0.443/0.614/0.509 | 0.967 / 0.800 |
+| **both (1.1.0)** | **0.200/0.286/0.238** | **0.314/0.471/0.383** | 0.443/0.614/0.509 | **0.967 / 0.833** |
 
 Ruler A is **unmoved by either bar**, and B by concentration. The whole cost is
 three questions of seventy at hit@1 on the warmest ruler, and six at hit@3.
@@ -323,20 +323,20 @@ from 0.314 and C to 0.329 from 0.443. **0.28 was chosen before this corpus
 existed and survived meeting it**, which is the only kind of evidence a default
 can have.
 
-**Latency** — warm corpus, 581 units `978a1d48a82a`, five consecutive
+**Latency** — warm corpus, 584 units `c9df00350cbd`, five consecutive
 invocations of `python -m benchmarks.query_latency --repeats 10` (420 samples
 each):
 
 | | | across the five |
 |---|---|---|
-| query, median | **1.0 ms** | 0.71 – 1.26 |
-| query, p95 | 1.7 ms | 1.2 – 3.7 |
-| refusing an unanswerable query | **0.029 ms** | 0.023 – 0.031 |
-| refusal cheaper than answering by | **~36×** | 31 – 44 |
+| query, median | **0.61 ms** | 0.60 – 0.64 |
+| query, p95 | 1.09 ms | 1.06 – 1.14 |
+| refusing an unanswerable query | **0.017 ms** | 0.015 – 0.019 |
+| refusal cheaper than answering by | **~36×** | 33 – 41 |
 
 Two significant figures and a spread, because that is the precision the
-measurement has. Across fifteen invocations over two releases on the same idle
-machine the median has landed anywhere from 0.62 to 1.44 ms and p95 from 1.09
+measurement has. Across twenty invocations over three releases on the same idle
+machine the median has landed anywhere from 0.51 to 1.44 ms and p95 from 0.85
 to 7.34 ms — a band wider than any change the code has ever made to this
 number. Releases before 1.3.0 published `0.83 ms / p95 1.68 ms` to three
 figures from a script that was never committed; both values sit inside that
@@ -391,17 +391,17 @@ framework reversed it. The honest claim is narrower than either table alone:
 | right file first | 22.9% | **37.1%** |
 | right file in top 3 | 45.7% | **57.1%** |
 | lines it hands back, all questions | 17,641 | — |
-| characters returned, all questions | 1,415,656 | **249,720** |
+| characters returned, all questions | 1,415,656 | **258,236** |
 | questions it answers | 30 | 30 |
 
 **Once the vocabulary exists, it is not close.**
 
-| this repository · 70 questions · 581 units `978a1d48a82a` · 304 described | Grep loop | rag-your-code |
+| this repository · 70 questions · 584 units `c9df00350cbd` · 304 described | Grep loop | rag-your-code |
 |---|---|---|
 | right file first | 22.9% | **58.6%** |
-| right file in top 3 | 54.3% | **75.7%** |
-| lines it hands back, all questions | 11,833 | — |
-| characters returned, all questions | 1,122,902 | **626,022** |
+| right file in top 3 | 54.3% | **77.1%** |
+| lines it hands back, all questions | 11,959 | — |
+| characters returned, all questions | 1,135,411 | **615,673** |
 | questions it answers | **61** | 60 |
 
 Those two tables are the whole argument of section 3.3, measured against a real
@@ -438,11 +438,17 @@ Four qualifications, because the table would otherwise flatter both sides:
   `the` gets every file back in no order. It is also why Grep declines nine of
   the seventy questions here — those had no word left that this corpus does not
   use everywhere.
-- **Payload is counted in characters on both sides.** On this repository Grep
-  hands back 18,400 characters per question it answers, unranked and without
-  spans, against 10,400 here, ranked and capped by `search.max_chars` — a
-  factor of 1.8. On Flask it is a factor of 5.7 — 47,200 characters against
-  8,300 — because a framework repeats its own vocabulary across many files and
+- **Payload is counted in characters on both sides**, and 1.4.1 changed what
+  fits in it. A generated description ends with the author's docstring so the
+  docstring is searchable, and the source printed below it said the same thing
+  again — 2,381 of 3,382 characters of prose header on Flask were a verbatim
+  repeat of the code beneath. The block no longer prints what the code shows,
+  and the same 12,000-character budget now carries **119 declarations instead
+  of 92** there, 323 instead of 305 here. Grep
+  hands back 18,600 characters per question it answers, unranked and without
+  spans, against 10,300 here, ranked and capped by `search.max_chars` — a
+  factor of 1.8. On Flask it is a factor of 5.5 — 47,200 characters against
+  8,600 — because a framework repeats its own vocabulary across many files and
   Grep has no way to rank what it finds. Both sides decline the same five of
   those 35 — and they are the five Chinese ones, all of them. A Chinese word is
   not a substring of English source and it is not a token in an index built

{rag_your_code-1.4.0 → rag_your_code-1.4.1}/pyproject.toml

@@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "rag-your-code"
-version = "1.4.0"
+version = "1.4.1"
 description = "A local, explainable RAG index for codebases and coding agents"
 readme = "README.md"
 requires-python = ">=3.10"

{rag_your_code-1.4.0 → rag_your_code-1.4.1/src/rag_your_code.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rag-your-code
-Version: 1.4.0
+Version: 1.4.1
 Summary: A local, explainable RAG index for codebases and coding agents
 Author: rag-your-code contributors
 License-Expression: MIT
@@ -295,11 +295,11 @@ repository had grown by ninety units.
 |---|---|---|---|---|---|
 | **A** Flask 3.1.3, no descriptions | what a first-time user gets | 35 | 0.200 | 0.286 | 0.238 |
 | **B** this repo, generated descriptions only | a cold index of familiar code | 70 | 0.314 | 0.471 | 0.383 |
-| **C** this repo, agent-written descriptions | the warmest case supported | 70 | 0.443 | 0.614 | 0.507 |
+| **C** this repo, agent-written descriptions | the warmest case supported | 70 | 0.443 | 0.614 | 0.509 |
 
 The corpora, without which none of the above is reproducible — **A** 1,572
-units, `5fd51169eacc`; **B** 581 units, `8e1e71942c1c`; **C** 581 units,
-`978a1d48a82a`. Ruler A grades **a copy of Flask 3.1.3 carried in this
+units, `5fd51169eacc`; **B** 584 units, `fb1f841fa43a`; **C** 584 units,
+`c9df00350cbd`. Ruler A grades **a copy of Flask 3.1.3 carried in this
 repository**, at [`benchmarks/corpus/flask`](benchmarks/corpus/flask), pinned
 to commit `22d9247`. Through 1.3.0 it graded a checkout on one machine, and
 that cost three things: two questions pointed at a declaration the subject had
@@ -330,10 +330,10 @@ Flask does not implement get through on exactly that.
 
 | gate | A hit@1/3/MRR | B hit@1/3/MRR | C hit@1/3/MRR | silence own / foreign |
 |---|---|---|---|---|
-| neither (pre-1.0.0) | 0.200/0.286/0.238 | 0.314/0.486/0.391 | 0.486/0.700/0.567 | 0.000 / 0.000 |
-| coverage only (1.0.0) | 0.200/0.286/0.238 | 0.314/0.471/0.383 | 0.486/0.686/0.562 | 0.600 / 0.733 |
-| concentration only | 0.200/0.286/0.238 | 0.314/0.471/0.383 | 0.443/0.614/0.507 | 0.967 / 0.800 |
-| **both (1.1.0)** | **0.200/0.286/0.238** | **0.314/0.471/0.383** | 0.443/0.614/0.507 | **0.967 / 0.833** |
+| neither (pre-1.0.0) | 0.200/0.286/0.238 | 0.314/0.486/0.391 | 0.486/0.700/0.569 | 0.000 / 0.000 |
+| coverage only (1.0.0) | 0.200/0.286/0.238 | 0.314/0.471/0.383 | 0.486/0.686/0.564 | 0.567 / 0.733 |
+| concentration only | 0.200/0.286/0.238 | 0.314/0.471/0.383 | 0.443/0.614/0.509 | 0.967 / 0.800 |
+| **both (1.1.0)** | **0.200/0.286/0.238** | **0.314/0.471/0.383** | 0.443/0.614/0.509 | **0.967 / 0.833** |
 
 Ruler A is **unmoved by either bar**, and B by concentration. The whole cost is
 three questions of seventy at hit@1 on the warmest ruler, and six at hit@3.
@@ -352,20 +352,20 @@ from 0.314 and C to 0.329 from 0.443. **0.28 was chosen before this corpus
 existed and survived meeting it**, which is the only kind of evidence a default
 can have.
 
-**Latency** — warm corpus, 581 units `978a1d48a82a`, five consecutive
+**Latency** — warm corpus, 584 units `c9df00350cbd`, five consecutive
 invocations of `python -m benchmarks.query_latency --repeats 10` (420 samples
 each):
 
 | | | across the five |
 |---|---|---|
-| query, median | **1.0 ms** | 0.71 – 1.26 |
-| query, p95 | 1.7 ms | 1.2 – 3.7 |
-| refusing an unanswerable query | **0.029 ms** | 0.023 – 0.031 |
-| refusal cheaper than answering by | **~36×** | 31 – 44 |
+| query, median | **0.61 ms** | 0.60 – 0.64 |
+| query, p95 | 1.09 ms | 1.06 – 1.14 |
+| refusing an unanswerable query | **0.017 ms** | 0.015 – 0.019 |
+| refusal cheaper than answering by | **~36×** | 33 – 41 |
 
 Two significant figures and a spread, because that is the precision the
-measurement has. Across fifteen invocations over two releases on the same idle
-machine the median has landed anywhere from 0.62 to 1.44 ms and p95 from 1.09
+measurement has. Across twenty invocations over three releases on the same idle
+machine the median has landed anywhere from 0.51 to 1.44 ms and p95 from 0.85
 to 7.34 ms — a band wider than any change the code has ever made to this
 number. Releases before 1.3.0 published `0.83 ms / p95 1.68 ms` to three
 figures from a script that was never committed; both values sit inside that
@@ -420,17 +420,17 @@ framework reversed it. The honest claim is narrower than either table alone:
 | right file first | 22.9% | **37.1%** |
 | right file in top 3 | 45.7% | **57.1%** |
 | lines it hands back, all questions | 17,641 | — |
-| characters returned, all questions | 1,415,656 | **249,720** |
+| characters returned, all questions | 1,415,656 | **258,236** |
 | questions it answers | 30 | 30 |
 
 **Once the vocabulary exists, it is not close.**
 
-| this repository · 70 questions · 581 units `978a1d48a82a` · 304 described | Grep loop | rag-your-code |
+| this repository · 70 questions · 584 units `c9df00350cbd` · 304 described | Grep loop | rag-your-code |
 |---|---|---|
 | right file first | 22.9% | **58.6%** |
-| right file in top 3 | 54.3% | **75.7%** |
-| lines it hands back, all questions | 11,833 | — |
-| characters returned, all questions | 1,122,902 | **626,022** |
+| right file in top 3 | 54.3% | **77.1%** |
+| lines it hands back, all questions | 11,959 | — |
+| characters returned, all questions | 1,135,411 | **615,673** |
 | questions it answers | **61** | 60 |
 
 Those two tables are the whole argument of section 3.3, measured against a real
@@ -467,11 +467,17 @@ Four qualifications, because the table would otherwise flatter both sides:
   `the` gets every file back in no order. It is also why Grep declines nine of
   the seventy questions here — those had no word left that this corpus does not
   use everywhere.
-- **Payload is counted in characters on both sides.** On this repository Grep
-  hands back 18,400 characters per question it answers, unranked and without
-  spans, against 10,400 here, ranked and capped by `search.max_chars` — a
-  factor of 1.8. On Flask it is a factor of 5.7 — 47,200 characters against
-  8,300 — because a framework repeats its own vocabulary across many files and
+- **Payload is counted in characters on both sides**, and 1.4.1 changed what
+  fits in it. A generated description ends with the author's docstring so the
+  docstring is searchable, and the source printed below it said the same thing
+  again — 2,381 of 3,382 characters of prose header on Flask were a verbatim
+  repeat of the code beneath. The block no longer prints what the code shows,
+  and the same 12,000-character budget now carries **119 declarations instead
+  of 92** there, 323 instead of 305 here. Grep
+  hands back 18,600 characters per question it answers, unranked and without
+  spans, against 10,300 here, ranked and capped by `search.max_chars` — a
+  factor of 1.8. On Flask it is a factor of 5.5 — 47,200 characters against
+  8,600 — because a framework repeats its own vocabulary across many files and
   Grep has no way to rank what it finds. Both sides decline the same five of
   those 35 — and they are the five Chinese ones, all of them. A Chinese word is
   not a substring of English source and it is not a token in an index built

{rag_your_code-1.4.0 → rag_your_code-1.4.1}/src/ragyourcode/__init__.py

@@ -3,4 +3,4 @@
 from .models import CodeUnit, SearchResult
 
 __all__ = ["CodeUnit", "SearchResult"]
-__version__ = "1.4.0"
+__version__ = "1.4.1"

{rag_your_code-1.4.0 → rag_your_code-1.4.1}/src/ragyourcode/annotate.py

@@ -5,6 +5,14 @@ from __future__ import annotations
 import ast
 import re
 
+# Where a generated description stops describing the signature and starts
+# quoting what the author wrote. Three modules needed to recognise it and all
+# three spelled it out: this one writes it, the parser writes it again for the
+# other fourteen languages, and `document` looked for it. A string literal
+# duplicated across modules that must agree is a rename away from a silent
+# disagreement.
+DOCUMENTED_MARKER = "Documented intent:"
+
 
 def _humanize(name: str) -> str:
     """Turns a programmer identifier into ordinary words: splits camelCase
@@ -37,7 +45,7 @@ def describe_python(node: ast.AST, source: str, calls: list[str], imports: list[
         pieces.append("using " + ", ".join(imports[:8]))
     doc = ast.get_docstring(node)
     if doc:
-        pieces.append("Documented intent: " + " ".join(doc.split()))
+        pieces.append(f"{DOCUMENTED_MARKER} " + " ".join(doc.split()))
     return ". ".join(pieces) + "."
 
 

{rag_your_code-1.4.0 → rag_your_code-1.4.1}/src/ragyourcode/document.py

@@ -26,6 +26,7 @@ import re
 from dataclasses import dataclass
 from pathlib import Path
 
+from .annotate import DOCUMENTED_MARKER
 from .descriptions import DescriptionStore
 from .models import CodeUnit
 from .parser import EXTENSIONS, parse_file
@@ -59,7 +60,6 @@ WRAP_WIDTH = 76
 # rewritten read as undocumented. That mistake proposed a hundred and seven
 # insertions against a repository with seventeen genuinely undocumented
 # declarations.
-DOCUMENTED_MARKER = "Documented intent:"
 _CJK = re.compile(r"[一-鿿]")
 
 

{rag_your_code-1.4.0 → rag_your_code-1.4.1}/src/ragyourcode/parser.py

@@ -26,7 +26,7 @@ import re
 from dataclasses import dataclass
 from pathlib import Path
 
-from .annotate import describe_python
+from .annotate import DOCUMENTED_MARKER, describe_python
 from .models import CodeUnit
 
 PARSER_VERSION = "3"
@@ -542,7 +542,7 @@ def _generic_units(path: Path, source: str, relative: str, language: str) -> lis
         if documented:
             # Phrased exactly as the Python path phrases a docstring, so the
             # two routes produce the same shape of text for the same thing.
-            description += f" Documented intent: {documented}"
+            description += f" {DOCUMENTED_MARKER} {documented}"
         parent = dotted.rsplit(".", 1)[0] if "." in dotted else None
         units.append(
             CodeUnit(

{rag_your_code-1.4.0 → rag_your_code-1.4.1}/src/ragyourcode/search.py

@@ -8,6 +8,7 @@ from bisect import bisect_left
 from collections import Counter, defaultdict
 from dataclasses import dataclass
 
+from .annotate import DOCUMENTED_MARKER
 from .config import BY_PATH
 from .embeddings import DEFAULT_DIMENSIONS, LocalEmbedder, tokenize
 from .models import CodeUnit, SearchResult
@@ -549,13 +550,47 @@ def search(
     ]
 
 
+def _visible_description(unit: CodeUnit) -> str:
+    """The description minus whatever the code printed under it already says.
+
+    A generated description ends with the author's own docstring, because that
+    is how a docstring becomes searchable -- and the source below the block
+    then prints it a second time. Measured on a repository whose author wrote
+    them, 2,381 of 3,382 characters of prose header were a verbatim repeat of
+    the code beneath it: a fifth of everything a query returned, paid for
+    twice. Retrieval is the half of RAG that has to fit in a context window,
+    so this is not a cosmetic saving.
+
+    Only the rendering drops it. ``searchable_text`` keeps the whole thing, so
+    no ruler moves and the docstring stays exactly as findable as it was.
+
+    The quoted docstring is re-flowed onto one line where the source has it
+    indented across many, so the test is against whitespace-collapsed source
+    rather than against the source itself. An authored description the source
+    does not carry is kept: it is the one part a reader cannot get from the
+    code.
+    """
+    head, marker, documented = unit.description.partition(DOCUMENTED_MARKER)
+    if not marker:
+        return unit.description
+    quoted = " ".join(documented.split())
+    # `describe_python` joins its pieces with ". " and closes with ".", so a
+    # docstring that already ended in one arrives here with two. Matching on
+    # the raw text found nothing at all on the first repository tried.
+    trimmed = quoted[:-1] if quoted.endswith(".") else quoted
+    body = " ".join(unit.source.split())
+    if quoted in body or trimmed in body:
+        return head.rstrip()
+    return unit.description
+
+
 def _block(result: SearchResult) -> str:
     """One result as an agent reads it: identifier, score, why it matched,
     what it is, and the code itself.
     """
     unit = result.unit
     evidence = "\nEvidence: " + " | ".join(result.evidence) if result.evidence else ""
-    return f"[{unit.id}] score={result.score:.3f}{evidence}\n{unit.description}\n```{unit.language}\n{unit.source}\n```"
+    return f"[{unit.id}] score={result.score:.3f}{evidence}\n{_visible_description(unit)}\n```{unit.language}\n{unit.source}\n```"
 
 
 def within_budget(results: list[SearchResult], max_chars: int) -> list[SearchResult]:

{rag_your_code-1.4.0 → rag_your_code-1.4.1}/tests/test_ranking.py

@@ -18,6 +18,7 @@ present anywhere in the unit -- ranks the *wrong* answer first.
 
 from __future__ import annotations
 
+import dataclasses
 from pathlib import Path
 
 from ragyourcode.indexer import build_units
@@ -200,3 +201,56 @@ def test_a_field_every_unit_leaves_empty_does_not_divide_by_zero():
     )
     index = build_search_index([unit])
     assert search([unit], "nothing", limit=1, search_index=index)[0].unit.id == unit.id
+
+
+def test_a_block_does_not_reprint_the_docstring_the_code_below_it_shows(tmp_path: Path):
+    """The author's own words reached the block twice: once quoted into the
+    generated description so they are searchable, once in the source.
+
+    Measured on a repository whose author wrote them, 2,381 of 3,382
+    characters of prose header were a verbatim repeat of the code beneath it.
+    At a fixed budget that is answers crowded out by their own duplicate: the
+    same twelve thousand characters carried 92 declarations before this and
+    119 after.
+
+    Both halves are asserted. Dropping the quote must not drop what only the
+    header says, and it must not touch what the index can find -- the docstring
+    stays in `searchable_text`, which is why no ruler moves.
+    """
+    (tmp_path / "billing.py").write_text(
+        'def charge(amount):\n'
+        '    """Charge the amount against the stored card and return a receipt."""\n'
+        '    return amount\n',
+        encoding="utf-8",
+    )
+    unit = build_units(tmp_path)[0]
+    index = build_search_index([unit])
+    rendered = _block(search([unit], "charge the amount", search_index=index, limit=1)[0])
+
+    quoted = "Charge the amount against the stored card and return a receipt."
+    assert rendered.count(quoted) == 1, "the docstring is printed once, by the code"
+    assert rendered.index(quoted) > rendered.index("```"), "the surviving copy is the one in the source"
+    assert "This method charge" in rendered, "the generated half is not in the source and must stay"
+    assert quoted in unit.searchable_text, "dropping it from the block must not drop it from the index"
+
+
+def test_a_written_description_the_source_does_not_carry_survives(tmp_path: Path):
+    """The rule is "the code already shows it", not "it came after the marker".
+
+    An authored description is the one part of a block a reader cannot recover
+    by reading the code, so a rule that keyed on the marker alone would delete
+    exactly the text this project spends tokens to produce.
+    """
+    (tmp_path / "billing.py").write_text(
+        'def charge(amount):\n'
+        '    """Charge the card."""\n'
+        '    return amount\n',
+        encoding="utf-8",
+    )
+    unit = build_units(tmp_path)[0]
+    unit = dataclasses.replace(
+        unit, description="This method charge. Documented intent: retries a declined authorisation once."
+    )
+    index = build_search_index([unit])
+    rendered = _block(search([unit], "charge the amount", search_index=index, limit=1)[0])
+    assert "retries a declined authorisation once." in rendered