rag-your-code 1.2.1__tar.gz → 1.4.0__tar.gz

{rag_your_code-1.2.1/src/rag_your_code.egg-info → rag_your_code-1.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rag-your-code
-Version: 1.2.1
+Version: 1.4.0
 Summary: A local, explainable RAG index for codebases and coding agents
 Author: rag-your-code contributors
 License-Expression: MIT
@@ -293,42 +293,87 @@ repository had grown by ninety units.
 
 | ruler | what it represents | n | hit@1 | hit@3 | MRR |
 |---|---|---|---|---|---|
-| **A** foreign repo, no descriptions | what a first-time user gets | 35 | 0.229 | 0.400 | 0.300 |
+| **A** Flask 3.1.3, no descriptions | what a first-time user gets | 35 | 0.200 | 0.286 | 0.238 |
 | **B** this repo, generated descriptions only | a cold index of familiar code | 70 | 0.314 | 0.471 | 0.383 |
 | **C** this repo, agent-written descriptions | the warmest case supported | 70 | 0.443 | 0.614 | 0.507 |
 
+The corpora, without which none of the above is reproducible — **A** 1,572
+units, `5fd51169eacc`; **B** 581 units, `8e1e71942c1c`; **C** 581 units,
+`978a1d48a82a`. Ruler A grades **a copy of Flask 3.1.3 carried in this
+repository**, at [`benchmarks/corpus/flask`](benchmarks/corpus/flask), pinned
+to commit `22d9247`. Through 1.3.0 it graded a checkout on one machine, and
+that cost three things: two questions pointed at a declaration the subject had
+renamed, a published score moved 0.257 → 0.229 with no code change because the
+subject had grown, and the model comparison below was taken against two
+different states of it. All three are now a `git clone` away from being
+checked, and CI runs this ruler as an ordinary job.
+
 **Refusal — the fourth ruler, 30 questions with no answer anywhere**
 
-| | this repo | foreign repo |
+| | this repo | Flask |
 |---|---|---|
-| correctly met with silence | **0.967** | **0.933** |
-| English only | **0.933** | **0.867** |
+| correctly met with silence | **0.967** | **0.833** |
+| English only | **0.933** | 0.667 |
 | Chinese only | **1.000** | **1.000** |
 | results resting on no lexical evidence, rulers A–C | **0.000** | **0.000** |
 
+Foreign silence fell from 0.933 to 0.833 when the subject changed, and the
+cause is a limit of the design rather than a defect. A word counts as evidence
+unless it occurs in more than 5% of units — a stopword list derived from the
+corpus, so that it needs no list and works in any language. Here `how`, `when`,
+`does` and `are` are everywhere, because 304 units carry written English prose.
+Across 1,572 units of mostly short, undocumented methods they occur in 1–5% of
+them and start counting as evidence. Five English questions about subjects
+Flask does not implement get through on exactly that.
+
 **What each bar costs and buys** — one corpus, gate varied alone:
 
 | gate | A hit@1/3/MRR | B hit@1/3/MRR | C hit@1/3/MRR | silence own / foreign |
 |---|---|---|---|---|
-| neither (pre-1.0.0) | 0.229/0.400/0.300 | 0.314/0.486/0.391 | 0.471/0.686/0.552 | 0.000 / 0.000 |
-| coverage only (1.0.0) | 0.229/0.400/0.300 | 0.314/0.471/0.383 | 0.471/0.671/0.548 | 0.700 / 0.767 |
-| **both (1.1.0)** | **0.229/0.400/0.300** | **0.314/0.471/0.383** | 0.443/0.614/0.507 | **0.967 / 0.933** |
-
-Rulers A and B are **identical to three decimals**. The entire cost is four
-questions of seventy on the warmest ruler. On these four rulers concentration
-subsumes coverage — stated plainly because it is true; coverage is kept because
-it answers a different question and names a different diagnosis.
-
-**Latency** — warm corpus, 557 units, 420 samples after warm-up:
-
-| | |
-|---|---|
-| query, median | **0.83 ms** |
-| query, p95 | 1.68 ms |
-| refusing an unanswerable query | **0.03 ms** |
-
-Refusal is cheaper than answering by a factor of forty: an unanswerable query
-touches only the posting lists of its own distinctive words, never the corpus.
+| neither (pre-1.0.0) | 0.200/0.286/0.238 | 0.314/0.486/0.391 | 0.486/0.700/0.567 | 0.000 / 0.000 |
+| coverage only (1.0.0) | 0.200/0.286/0.238 | 0.314/0.471/0.383 | 0.486/0.686/0.562 | 0.600 / 0.733 |
+| concentration only | 0.200/0.286/0.238 | 0.314/0.471/0.383 | 0.443/0.614/0.507 | 0.967 / 0.800 |
+| **both (1.1.0)** | **0.200/0.286/0.238** | **0.314/0.471/0.383** | 0.443/0.614/0.507 | **0.967 / 0.833** |
+
+Ruler A is **unmoved by either bar**, and B by concentration. The whole cost is
+three questions of seventy at hit@1 on the warmest ruler, and six at hit@3.
+
+Through 1.3.0 this section said concentration subsumes coverage. **On a corpus
+this project did not choose, it does not.** Both bars together silence 0.833 of
+the foreign absent questions, against 0.800 for concentration alone and 0.733
+for coverage alone. One question — and the first time in four releases that
+keeping both has been worth a measurable amount rather than worth a different
+diagnosis.
+
+Raising the concentration bar buys the remaining silence, and is refused,
+because it is bought out of the answers: at 0.50 the foreign absent ruler is
+silent on all thirty while ruler A falls to 0.086 hit@1 from 0.200, B to 0.214
+from 0.314 and C to 0.329 from 0.443. **0.28 was chosen before this corpus
+existed and survived meeting it**, which is the only kind of evidence a default
+can have.
+
+**Latency** — warm corpus, 581 units `978a1d48a82a`, five consecutive
+invocations of `python -m benchmarks.query_latency --repeats 10` (420 samples
+each):
+
+| | | across the five |
+|---|---|---|
+| query, median | **1.0 ms** | 0.71 – 1.26 |
+| query, p95 | 1.7 ms | 1.2 – 3.7 |
+| refusing an unanswerable query | **0.029 ms** | 0.023 – 0.031 |
+| refusal cheaper than answering by | **~36×** | 31 – 44 |
+
+Two significant figures and a spread, because that is the precision the
+measurement has. Across fifteen invocations over two releases on the same idle
+machine the median has landed anywhere from 0.62 to 1.44 ms and p95 from 1.09
+to 7.34 ms — a band wider than any change the code has ever made to this
+number. Releases before 1.3.0 published `0.83 ms / p95 1.68 ms` to three
+figures from a script that was never committed; both values sit inside that
+band, which is the point: they were unfalsifiable rather than wrong.
+
+Refusal is cheap for a structural reason, not a tuned one: an unanswerable
+query touches only the posting lists of its own distinctive words, and never
+reaches ranking at all.
 
 **Scale**, synthetic 10,000-unit repository (500 files):
 
@@ -350,8 +395,11 @@ touches only the posting lists of its own distinctive words, never the corpus.
 | with a usable signature | **91 / 91** |
 | units invented that do not exist | **0** |
 
-Directional local measurements, not service levels. Reproduce with
-`python benchmarks/repo_queries.py` and `python benchmarks/large_repo.py`.
+Directional local measurements, not service levels — but every one of them is
+now a command rather than a memory, which two of them were not before. Each
+prints the corpus fingerprint beside its score; quote both or neither.
+[`benchmarks/README.md`](benchmarks/README.md) lists the six scripts and what
+each is for, and the corpus one of them grades is now carried here too.
 
 ## 7 · `rag-your-code search` vs a Grep loop
 
@@ -361,45 +409,74 @@ many hit. That is what this reproduces — same corpus, same questions, same
 ruler, scored at **file** granularity so Grep is not penalised for lacking
 declaration spans.
 
-**On a repository nobody has described, Grep wins.** That is the measured
-result and it is not softened here.
+**Which side wins on an undescribed repository depends on the repository.**
+Through 1.3.0 this section said flatly that Grep wins there, because the one
+undescribed repository ever measured was a hook-heavy tool whose questions were
+answerable by matching identifiers. Swapping the subject for a public web
+framework reversed it. The honest claim is narrower than either table alone:
 
-| foreign repository · 35 questions · 1,267 units · no descriptions | Grep loop | rag-your-code |
+| Flask 3.1.3 · 35 questions · 1,572 units `5fd51169eacc` · no descriptions | Grep loop | rag-your-code |
 |---|---|---|
-| right file first | **34.3%** | 31.4% |
-| right file in top 3 | **60.0%** | 48.6% |
-| lines matched across the repo, all questions | 33,213 | — |
-| characters returned, all questions | — | **163,294** |
-| questions it answers | 35 | 28 |
+| right file first | 22.9% | **37.1%** |
+| right file in top 3 | 45.7% | **57.1%** |
+| lines it hands back, all questions | 17,641 | — |
+| characters returned, all questions | 1,415,656 | **249,720** |
+| questions it answers | 30 | 30 |
 
 **Once the vocabulary exists, it is not close.**
 
-| this repository · 70 questions · 569 units · 304 described | Grep loop | rag-your-code |
+| this repository · 70 questions · 581 units `978a1d48a82a` · 304 described | Grep loop | rag-your-code |
 |---|---|---|
-| right file first | 25.7% | **58.6%** |
-| right file in top 3 | 64.3% | **75.7%** |
-| lines matched across the repo, all questions | 40,150 | — |
-| characters returned, all questions | — | **277,327** |
-| questions it answers | 70 | 60 |
+| right file first | 22.9% | **58.6%** |
+| right file in top 3 | 54.3% | **75.7%** |
+| lines it hands back, all questions | 11,833 | — |
+| characters returned, all questions | 1,122,902 | **626,022** |
+| questions it answers | **61** | 60 |
 
 Those two tables are the whole argument of section 3.3, measured against a real
 baseline instead of asserted. A cold index retrieves against a sentence the
-parser generated from identifiers the author already chose — so it is competing
-with Grep using Grep's own information, and losing, because Grep does not have
-to guess which of the matching files is the definition. Descriptions put words
-in the index that the source never contained, and first-place accuracy goes
-from below Grep's to **more than double** it.
-
-Three qualifications, because the table would otherwise flatter both sides:
+parser generated from identifiers the author already chose, plus whatever
+docstrings the author wrote — so how it fares against Grep is decided by how
+much prose the repository already contains. Flask has a written docstring on
+most public methods, and the cold index beats Grep there without a single
+description being added. On the previous subject, a tool with terse comments
+and long identifiers, the same cold index lost to Grep by the same margin.
+
+What does not depend on the subject is what descriptions buy: on this
+repository first-place accuracy goes to **more than double** Grep's, and the
+payload comes back ranked, spanned, and roughly half the size.
+
+**Both tables come from `python -m benchmarks.grep_baseline`**, which is what
+changed in 1.3.0. Until then this section — the strongest claim the project
+makes — was published from a script that had never been committed, so nothing
+here could be checked and the word "Grep loop" had no precise meaning. The
+committed version defines it: take the query's words, drop the ones the corpus
+itself shows are everywhere, run one substring search per remaining word over
+exactly the files the index was built from, rank each file by how many distinct
+words hit it, break ties on path. Reconstructing it reproduced this side's
+figures exactly and moved Grep's, which is the expected shape — the ranked arm
+was always a call into shipped code, and the baseline never was.
+
+Four qualifications, because the table would otherwise flatter both sides:
 
 - **Scored at file granularity**, which understates this side. A Grep hit is a
   file; a hit here is a declaration with an exact span, a score, and the words
   it matched on. The agent that reads the result opens 40 lines, not a file.
-- **Grep answers everything.** It never declines, which is why it hands back
-  33,213 matching lines for 35 questions — about 950 lines per question, no
-  ranking, no spans, no indication which match is the definition. This returns
-  roughly 5,800 characters per question, ranked. Seven of 35 and ten of 70
-  questions come back empty here instead, with a reason.
+- **Dropping the corpus-common words is generous to Grep**, and it is what
+  makes the baseline a fair one rather than a straw man: an agent that greps
+  `the` gets every file back in no order. It is also why Grep declines nine of
+  the seventy questions here — those had no word left that this corpus does not
+  use everywhere.
+- **Payload is counted in characters on both sides.** On this repository Grep
+  hands back 18,400 characters per question it answers, unranked and without
+  spans, against 10,400 here, ranked and capped by `search.max_chars` — a
+  factor of 1.8. On Flask it is a factor of 5.7 — 47,200 characters against
+  8,300 — because a framework repeats its own vocabulary across many files and
+  Grep has no way to rank what it finds. Both sides decline the same five of
+  those 35 — and they are the five Chinese ones, all of them. A Chinese word is
+  not a substring of English source and it is not a token in an index built
+  from English source, so on a repository written in one language the cold
+  cross-language case is not this tool's failure but the corpus's.
 - **Grep wins outright when you know the string.** `grep -rn "COMMON_TERM"` is
   exact, instant and complete, and nothing here replaces it.
 
@@ -464,19 +541,23 @@ The extra is optional by construction: `dependencies = []` is what a default
 install gets, the import happens inside the constructor, and a test asserts the
 default provider imports none of it.
 
-**Measured, on the same four rulers.** This is the first release with a real
-model behind these numbers — 0.8.0 shipped the seam and said plainly that its
-benefit was unmeasured.
+**Measured on the same four rulers, both arms against one corpus.** 1.1.0
+published this comparison and read it as a win. Its largest gain was on the
+foreign ruler, whose two arms turned out to have been taken against two
+different states of a repository being edited while the script ran. Repeated
+against a pinned corpus:
 
-| ruler | signed hash (default) | MiniLM, local |
-|---|---|---|
-| **A** foreign, cold | 0.229 / 0.400 / 0.300 | **0.286 / 0.457 / 0.357** |
-| **B** own, cold | 0.314 / 0.471 / 0.383 | **0.329 / 0.486 / 0.400** |
-| **C** own, described | 0.443 / 0.614 / 0.507 | 0.443 / **0.671 / 0.540** |
-| **D** silence, own / foreign | 0.967 / 0.933 | 0.967 / 0.933 |
+| ruler | corpus | signed hash (default) | MiniLM, local |
+|---|---|---|---|
+| **A** foreign, cold | 1,572 `5fd51169eacc` | **0.200 / 0.286 / 0.238** | 0.171 / 0.257 / 0.214 |
+| **B** own, cold | 581 `8e1e71942c1c` | 0.314 / 0.471 / 0.383 | 0.314 / 0.471 / 0.383 |
+| **C** own, described | 581 `978a1d48a82a` | **0.443 / 0.614 / 0.507** | 0.429 / 0.600 / 0.500 |
+| **D** silence, own / foreign | as above | 0.967 / 0.833 | 0.967 / 0.833 |
 
-Better on every positive ruler, with refusal unchanged. The pairs the hash
-scores exactly zero:
+**Worse or identical on every ruler.** It is shipped anyway, as an extra nobody
+has to install, because it does one thing the hash cannot do at all and these
+rulers cannot see: reach a unit that shares no word with the question. The
+pairs the hash scores exactly zero:
 
 | pair | signed hash | MiniLM |
 |---|---|---|
@@ -494,6 +575,10 @@ a threshold on a score and the distributions overlap (0.469 vs 0.418 median),
 and a scale-free standout metric took ruler B from 0.329 to 0.186 for two
 thirds of the silence. Applying the lexical bars costs ruler A nothing.
 
+If you install it expecting the hit rates above to move, they will not. Install
+it for the cross-language and paraphrase cases in the table above, which is
+where the difference between the two columns actually lives.
+
 **A hosted endpoint** is the third option, and the only one that sends your
 source anywhere:
 

{rag_your_code-1.2.1 → rag_your_code-1.4.0}/README.md

@@ -264,42 +264,87 @@ repository had grown by ninety units.
 
 | ruler | what it represents | n | hit@1 | hit@3 | MRR |
 |---|---|---|---|---|---|
-| **A** foreign repo, no descriptions | what a first-time user gets | 35 | 0.229 | 0.400 | 0.300 |
+| **A** Flask 3.1.3, no descriptions | what a first-time user gets | 35 | 0.200 | 0.286 | 0.238 |
 | **B** this repo, generated descriptions only | a cold index of familiar code | 70 | 0.314 | 0.471 | 0.383 |
 | **C** this repo, agent-written descriptions | the warmest case supported | 70 | 0.443 | 0.614 | 0.507 |
 
+The corpora, without which none of the above is reproducible — **A** 1,572
+units, `5fd51169eacc`; **B** 581 units, `8e1e71942c1c`; **C** 581 units,
+`978a1d48a82a`. Ruler A grades **a copy of Flask 3.1.3 carried in this
+repository**, at [`benchmarks/corpus/flask`](benchmarks/corpus/flask), pinned
+to commit `22d9247`. Through 1.3.0 it graded a checkout on one machine, and
+that cost three things: two questions pointed at a declaration the subject had
+renamed, a published score moved 0.257 → 0.229 with no code change because the
+subject had grown, and the model comparison below was taken against two
+different states of it. All three are now a `git clone` away from being
+checked, and CI runs this ruler as an ordinary job.
+
 **Refusal — the fourth ruler, 30 questions with no answer anywhere**
 
-| | this repo | foreign repo |
+| | this repo | Flask |
 |---|---|---|
-| correctly met with silence | **0.967** | **0.933** |
-| English only | **0.933** | **0.867** |
+| correctly met with silence | **0.967** | **0.833** |
+| English only | **0.933** | 0.667 |
 | Chinese only | **1.000** | **1.000** |
 | results resting on no lexical evidence, rulers A–C | **0.000** | **0.000** |
 
+Foreign silence fell from 0.933 to 0.833 when the subject changed, and the
+cause is a limit of the design rather than a defect. A word counts as evidence
+unless it occurs in more than 5% of units — a stopword list derived from the
+corpus, so that it needs no list and works in any language. Here `how`, `when`,
+`does` and `are` are everywhere, because 304 units carry written English prose.
+Across 1,572 units of mostly short, undocumented methods they occur in 1–5% of
+them and start counting as evidence. Five English questions about subjects
+Flask does not implement get through on exactly that.
+
 **What each bar costs and buys** — one corpus, gate varied alone:
 
 | gate | A hit@1/3/MRR | B hit@1/3/MRR | C hit@1/3/MRR | silence own / foreign |
 |---|---|---|---|---|
-| neither (pre-1.0.0) | 0.229/0.400/0.300 | 0.314/0.486/0.391 | 0.471/0.686/0.552 | 0.000 / 0.000 |
-| coverage only (1.0.0) | 0.229/0.400/0.300 | 0.314/0.471/0.383 | 0.471/0.671/0.548 | 0.700 / 0.767 |
-| **both (1.1.0)** | **0.229/0.400/0.300** | **0.314/0.471/0.383** | 0.443/0.614/0.507 | **0.967 / 0.933** |
-
-Rulers A and B are **identical to three decimals**. The entire cost is four
-questions of seventy on the warmest ruler. On these four rulers concentration
-subsumes coverage — stated plainly because it is true; coverage is kept because
-it answers a different question and names a different diagnosis.
-
-**Latency** — warm corpus, 557 units, 420 samples after warm-up:
-
-| | |
-|---|---|
-| query, median | **0.83 ms** |
-| query, p95 | 1.68 ms |
-| refusing an unanswerable query | **0.03 ms** |
-
-Refusal is cheaper than answering by a factor of forty: an unanswerable query
-touches only the posting lists of its own distinctive words, never the corpus.
+| neither (pre-1.0.0) | 0.200/0.286/0.238 | 0.314/0.486/0.391 | 0.486/0.700/0.567 | 0.000 / 0.000 |
+| coverage only (1.0.0) | 0.200/0.286/0.238 | 0.314/0.471/0.383 | 0.486/0.686/0.562 | 0.600 / 0.733 |
+| concentration only | 0.200/0.286/0.238 | 0.314/0.471/0.383 | 0.443/0.614/0.507 | 0.967 / 0.800 |
+| **both (1.1.0)** | **0.200/0.286/0.238** | **0.314/0.471/0.383** | 0.443/0.614/0.507 | **0.967 / 0.833** |
+
+Ruler A is **unmoved by either bar**, and B by concentration. The whole cost is
+three questions of seventy at hit@1 on the warmest ruler, and six at hit@3.
+
+Through 1.3.0 this section said concentration subsumes coverage. **On a corpus
+this project did not choose, it does not.** Both bars together silence 0.833 of
+the foreign absent questions, against 0.800 for concentration alone and 0.733
+for coverage alone. One question — and the first time in four releases that
+keeping both has been worth a measurable amount rather than worth a different
+diagnosis.
+
+Raising the concentration bar buys the remaining silence, and is refused,
+because it is bought out of the answers: at 0.50 the foreign absent ruler is
+silent on all thirty while ruler A falls to 0.086 hit@1 from 0.200, B to 0.214
+from 0.314 and C to 0.329 from 0.443. **0.28 was chosen before this corpus
+existed and survived meeting it**, which is the only kind of evidence a default
+can have.
+
+**Latency** — warm corpus, 581 units `978a1d48a82a`, five consecutive
+invocations of `python -m benchmarks.query_latency --repeats 10` (420 samples
+each):
+
+| | | across the five |
+|---|---|---|
+| query, median | **1.0 ms** | 0.71 – 1.26 |
+| query, p95 | 1.7 ms | 1.2 – 3.7 |
+| refusing an unanswerable query | **0.029 ms** | 0.023 – 0.031 |
+| refusal cheaper than answering by | **~36×** | 31 – 44 |
+
+Two significant figures and a spread, because that is the precision the
+measurement has. Across fifteen invocations over two releases on the same idle
+machine the median has landed anywhere from 0.62 to 1.44 ms and p95 from 1.09
+to 7.34 ms — a band wider than any change the code has ever made to this
+number. Releases before 1.3.0 published `0.83 ms / p95 1.68 ms` to three
+figures from a script that was never committed; both values sit inside that
+band, which is the point: they were unfalsifiable rather than wrong.
+
+Refusal is cheap for a structural reason, not a tuned one: an unanswerable
+query touches only the posting lists of its own distinctive words, and never
+reaches ranking at all.
 
 **Scale**, synthetic 10,000-unit repository (500 files):
 
@@ -321,8 +366,11 @@ touches only the posting lists of its own distinctive words, never the corpus.
 | with a usable signature | **91 / 91** |
 | units invented that do not exist | **0** |
 
-Directional local measurements, not service levels. Reproduce with
-`python benchmarks/repo_queries.py` and `python benchmarks/large_repo.py`.
+Directional local measurements, not service levels — but every one of them is
+now a command rather than a memory, which two of them were not before. Each
+prints the corpus fingerprint beside its score; quote both or neither.
+[`benchmarks/README.md`](benchmarks/README.md) lists the six scripts and what
+each is for, and the corpus one of them grades is now carried here too.
 
 ## 7 · `rag-your-code search` vs a Grep loop
 
@@ -332,45 +380,74 @@ many hit. That is what this reproduces — same corpus, same questions, same
 ruler, scored at **file** granularity so Grep is not penalised for lacking
 declaration spans.
 
-**On a repository nobody has described, Grep wins.** That is the measured
-result and it is not softened here.
+**Which side wins on an undescribed repository depends on the repository.**
+Through 1.3.0 this section said flatly that Grep wins there, because the one
+undescribed repository ever measured was a hook-heavy tool whose questions were
+answerable by matching identifiers. Swapping the subject for a public web
+framework reversed it. The honest claim is narrower than either table alone:
 
-| foreign repository · 35 questions · 1,267 units · no descriptions | Grep loop | rag-your-code |
+| Flask 3.1.3 · 35 questions · 1,572 units `5fd51169eacc` · no descriptions | Grep loop | rag-your-code |
 |---|---|---|
-| right file first | **34.3%** | 31.4% |
-| right file in top 3 | **60.0%** | 48.6% |
-| lines matched across the repo, all questions | 33,213 | — |
-| characters returned, all questions | — | **163,294** |
-| questions it answers | 35 | 28 |
+| right file first | 22.9% | **37.1%** |
+| right file in top 3 | 45.7% | **57.1%** |
+| lines it hands back, all questions | 17,641 | — |
+| characters returned, all questions | 1,415,656 | **249,720** |
+| questions it answers | 30 | 30 |
 
 **Once the vocabulary exists, it is not close.**
 
-| this repository · 70 questions · 569 units · 304 described | Grep loop | rag-your-code |
+| this repository · 70 questions · 581 units `978a1d48a82a` · 304 described | Grep loop | rag-your-code |
 |---|---|---|
-| right file first | 25.7% | **58.6%** |
-| right file in top 3 | 64.3% | **75.7%** |
-| lines matched across the repo, all questions | 40,150 | — |
-| characters returned, all questions | — | **277,327** |
-| questions it answers | 70 | 60 |
+| right file first | 22.9% | **58.6%** |
+| right file in top 3 | 54.3% | **75.7%** |
+| lines it hands back, all questions | 11,833 | — |
+| characters returned, all questions | 1,122,902 | **626,022** |
+| questions it answers | **61** | 60 |
 
 Those two tables are the whole argument of section 3.3, measured against a real
 baseline instead of asserted. A cold index retrieves against a sentence the
-parser generated from identifiers the author already chose — so it is competing
-with Grep using Grep's own information, and losing, because Grep does not have
-to guess which of the matching files is the definition. Descriptions put words
-in the index that the source never contained, and first-place accuracy goes
-from below Grep's to **more than double** it.
-
-Three qualifications, because the table would otherwise flatter both sides:
+parser generated from identifiers the author already chose, plus whatever
+docstrings the author wrote — so how it fares against Grep is decided by how
+much prose the repository already contains. Flask has a written docstring on
+most public methods, and the cold index beats Grep there without a single
+description being added. On the previous subject, a tool with terse comments
+and long identifiers, the same cold index lost to Grep by the same margin.
+
+What does not depend on the subject is what descriptions buy: on this
+repository first-place accuracy goes to **more than double** Grep's, and the
+payload comes back ranked, spanned, and roughly half the size.
+
+**Both tables come from `python -m benchmarks.grep_baseline`**, which is what
+changed in 1.3.0. Until then this section — the strongest claim the project
+makes — was published from a script that had never been committed, so nothing
+here could be checked and the word "Grep loop" had no precise meaning. The
+committed version defines it: take the query's words, drop the ones the corpus
+itself shows are everywhere, run one substring search per remaining word over
+exactly the files the index was built from, rank each file by how many distinct
+words hit it, break ties on path. Reconstructing it reproduced this side's
+figures exactly and moved Grep's, which is the expected shape — the ranked arm
+was always a call into shipped code, and the baseline never was.
+
+Four qualifications, because the table would otherwise flatter both sides:
 
 - **Scored at file granularity**, which understates this side. A Grep hit is a
   file; a hit here is a declaration with an exact span, a score, and the words
   it matched on. The agent that reads the result opens 40 lines, not a file.
-- **Grep answers everything.** It never declines, which is why it hands back
-  33,213 matching lines for 35 questions — about 950 lines per question, no
-  ranking, no spans, no indication which match is the definition. This returns
-  roughly 5,800 characters per question, ranked. Seven of 35 and ten of 70
-  questions come back empty here instead, with a reason.
+- **Dropping the corpus-common words is generous to Grep**, and it is what
+  makes the baseline a fair one rather than a straw man: an agent that greps
+  `the` gets every file back in no order. It is also why Grep declines nine of
+  the seventy questions here — those had no word left that this corpus does not
+  use everywhere.
+- **Payload is counted in characters on both sides.** On this repository Grep
+  hands back 18,400 characters per question it answers, unranked and without
+  spans, against 10,400 here, ranked and capped by `search.max_chars` — a
+  factor of 1.8. On Flask it is a factor of 5.7 — 47,200 characters against
+  8,300 — because a framework repeats its own vocabulary across many files and
+  Grep has no way to rank what it finds. Both sides decline the same five of
+  those 35 — and they are the five Chinese ones, all of them. A Chinese word is
+  not a substring of English source and it is not a token in an index built
+  from English source, so on a repository written in one language the cold
+  cross-language case is not this tool's failure but the corpus's.
 - **Grep wins outright when you know the string.** `grep -rn "COMMON_TERM"` is
   exact, instant and complete, and nothing here replaces it.
 
@@ -435,19 +512,23 @@ The extra is optional by construction: `dependencies = []` is what a default
 install gets, the import happens inside the constructor, and a test asserts the
 default provider imports none of it.
 
-**Measured, on the same four rulers.** This is the first release with a real
-model behind these numbers — 0.8.0 shipped the seam and said plainly that its
-benefit was unmeasured.
+**Measured on the same four rulers, both arms against one corpus.** 1.1.0
+published this comparison and read it as a win. Its largest gain was on the
+foreign ruler, whose two arms turned out to have been taken against two
+different states of a repository being edited while the script ran. Repeated
+against a pinned corpus:
 
-| ruler | signed hash (default) | MiniLM, local |
-|---|---|---|
-| **A** foreign, cold | 0.229 / 0.400 / 0.300 | **0.286 / 0.457 / 0.357** |
-| **B** own, cold | 0.314 / 0.471 / 0.383 | **0.329 / 0.486 / 0.400** |
-| **C** own, described | 0.443 / 0.614 / 0.507 | 0.443 / **0.671 / 0.540** |
-| **D** silence, own / foreign | 0.967 / 0.933 | 0.967 / 0.933 |
+| ruler | corpus | signed hash (default) | MiniLM, local |
+|---|---|---|---|
+| **A** foreign, cold | 1,572 `5fd51169eacc` | **0.200 / 0.286 / 0.238** | 0.171 / 0.257 / 0.214 |
+| **B** own, cold | 581 `8e1e71942c1c` | 0.314 / 0.471 / 0.383 | 0.314 / 0.471 / 0.383 |
+| **C** own, described | 581 `978a1d48a82a` | **0.443 / 0.614 / 0.507** | 0.429 / 0.600 / 0.500 |
+| **D** silence, own / foreign | as above | 0.967 / 0.833 | 0.967 / 0.833 |
 
-Better on every positive ruler, with refusal unchanged. The pairs the hash
-scores exactly zero:
+**Worse or identical on every ruler.** It is shipped anyway, as an extra nobody
+has to install, because it does one thing the hash cannot do at all and these
+rulers cannot see: reach a unit that shares no word with the question. The
+pairs the hash scores exactly zero:
 
 | pair | signed hash | MiniLM |
 |---|---|---|
@@ -465,6 +546,10 @@ a threshold on a score and the distributions overlap (0.469 vs 0.418 median),
 and a scale-free standout metric took ruler B from 0.329 to 0.186 for two
 thirds of the silence. Applying the lexical bars costs ruler A nothing.
 
+If you install it expecting the hit rates above to move, they will not. Install
+it for the cross-language and paraphrase cases in the table above, which is
+where the difference between the two columns actually lives.
+
 **A hosted endpoint** is the third option, and the only one that sends your
 source anywhere:
 

{rag_your_code-1.2.1 → rag_your_code-1.4.0}/pyproject.toml

@@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "rag-your-code"
-version = "1.2.1"
+version = "1.4.0"
 description = "A local, explainable RAG index for codebases and coding agents"
 readme = "README.md"
 requires-python = ">=3.10"

{rag_your_code-1.2.1 → rag_your_code-1.4.0/src/rag_your_code.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rag-your-code
-Version: 1.2.1
+Version: 1.4.0
 Summary: A local, explainable RAG index for codebases and coding agents
 Author: rag-your-code contributors
 License-Expression: MIT
@@ -293,42 +293,87 @@ repository had grown by ninety units.
 
 | ruler | what it represents | n | hit@1 | hit@3 | MRR |
 |---|---|---|---|---|---|
-| **A** foreign repo, no descriptions | what a first-time user gets | 35 | 0.229 | 0.400 | 0.300 |
+| **A** Flask 3.1.3, no descriptions | what a first-time user gets | 35 | 0.200 | 0.286 | 0.238 |
 | **B** this repo, generated descriptions only | a cold index of familiar code | 70 | 0.314 | 0.471 | 0.383 |
 | **C** this repo, agent-written descriptions | the warmest case supported | 70 | 0.443 | 0.614 | 0.507 |
 
+The corpora, without which none of the above is reproducible — **A** 1,572
+units, `5fd51169eacc`; **B** 581 units, `8e1e71942c1c`; **C** 581 units,
+`978a1d48a82a`. Ruler A grades **a copy of Flask 3.1.3 carried in this
+repository**, at [`benchmarks/corpus/flask`](benchmarks/corpus/flask), pinned
+to commit `22d9247`. Through 1.3.0 it graded a checkout on one machine, and
+that cost three things: two questions pointed at a declaration the subject had
+renamed, a published score moved 0.257 → 0.229 with no code change because the
+subject had grown, and the model comparison below was taken against two
+different states of it. All three are now a `git clone` away from being
+checked, and CI runs this ruler as an ordinary job.
+
 **Refusal — the fourth ruler, 30 questions with no answer anywhere**
 
-| | this repo | foreign repo |
+| | this repo | Flask |
 |---|---|---|
-| correctly met with silence | **0.967** | **0.933** |
-| English only | **0.933** | **0.867** |
+| correctly met with silence | **0.967** | **0.833** |
+| English only | **0.933** | 0.667 |
 | Chinese only | **1.000** | **1.000** |
 | results resting on no lexical evidence, rulers A–C | **0.000** | **0.000** |
 
+Foreign silence fell from 0.933 to 0.833 when the subject changed, and the
+cause is a limit of the design rather than a defect. A word counts as evidence
+unless it occurs in more than 5% of units — a stopword list derived from the
+corpus, so that it needs no list and works in any language. Here `how`, `when`,
+`does` and `are` are everywhere, because 304 units carry written English prose.
+Across 1,572 units of mostly short, undocumented methods they occur in 1–5% of
+them and start counting as evidence. Five English questions about subjects
+Flask does not implement get through on exactly that.
+
 **What each bar costs and buys** — one corpus, gate varied alone:
 
 | gate | A hit@1/3/MRR | B hit@1/3/MRR | C hit@1/3/MRR | silence own / foreign |
 |---|---|---|---|---|
-| neither (pre-1.0.0) | 0.229/0.400/0.300 | 0.314/0.486/0.391 | 0.471/0.686/0.552 | 0.000 / 0.000 |
-| coverage only (1.0.0) | 0.229/0.400/0.300 | 0.314/0.471/0.383 | 0.471/0.671/0.548 | 0.700 / 0.767 |
-| **both (1.1.0)** | **0.229/0.400/0.300** | **0.314/0.471/0.383** | 0.443/0.614/0.507 | **0.967 / 0.933** |
-
-Rulers A and B are **identical to three decimals**. The entire cost is four
-questions of seventy on the warmest ruler. On these four rulers concentration
-subsumes coverage — stated plainly because it is true; coverage is kept because
-it answers a different question and names a different diagnosis.
-
-**Latency** — warm corpus, 557 units, 420 samples after warm-up:
-
-| | |
-|---|---|
-| query, median | **0.83 ms** |
-| query, p95 | 1.68 ms |
-| refusing an unanswerable query | **0.03 ms** |
-
-Refusal is cheaper than answering by a factor of forty: an unanswerable query
-touches only the posting lists of its own distinctive words, never the corpus.
+| neither (pre-1.0.0) | 0.200/0.286/0.238 | 0.314/0.486/0.391 | 0.486/0.700/0.567 | 0.000 / 0.000 |
+| coverage only (1.0.0) | 0.200/0.286/0.238 | 0.314/0.471/0.383 | 0.486/0.686/0.562 | 0.600 / 0.733 |
+| concentration only | 0.200/0.286/0.238 | 0.314/0.471/0.383 | 0.443/0.614/0.507 | 0.967 / 0.800 |
+| **both (1.1.0)** | **0.200/0.286/0.238** | **0.314/0.471/0.383** | 0.443/0.614/0.507 | **0.967 / 0.833** |
+
+Ruler A is **unmoved by either bar**, and B by concentration. The whole cost is
+three questions of seventy at hit@1 on the warmest ruler, and six at hit@3.
+
+Through 1.3.0 this section said concentration subsumes coverage. **On a corpus
+this project did not choose, it does not.** Both bars together silence 0.833 of
+the foreign absent questions, against 0.800 for concentration alone and 0.733
+for coverage alone. One question — and the first time in four releases that
+keeping both has been worth a measurable amount rather than worth a different
+diagnosis.
+
+Raising the concentration bar buys the remaining silence, and is refused,
+because it is bought out of the answers: at 0.50 the foreign absent ruler is
+silent on all thirty while ruler A falls to 0.086 hit@1 from 0.200, B to 0.214
+from 0.314 and C to 0.329 from 0.443. **0.28 was chosen before this corpus
+existed and survived meeting it**, which is the only kind of evidence a default
+can have.
+
+**Latency** — warm corpus, 581 units `978a1d48a82a`, five consecutive
+invocations of `python -m benchmarks.query_latency --repeats 10` (420 samples
+each):
+
+| | | across the five |
+|---|---|---|
+| query, median | **1.0 ms** | 0.71 – 1.26 |
+| query, p95 | 1.7 ms | 1.2 – 3.7 |
+| refusing an unanswerable query | **0.029 ms** | 0.023 – 0.031 |
+| refusal cheaper than answering by | **~36×** | 31 – 44 |
+
+Two significant figures and a spread, because that is the precision the
+measurement has. Across fifteen invocations over two releases on the same idle
+machine the median has landed anywhere from 0.62 to 1.44 ms and p95 from 1.09
+to 7.34 ms — a band wider than any change the code has ever made to this
+number. Releases before 1.3.0 published `0.83 ms / p95 1.68 ms` to three
+figures from a script that was never committed; both values sit inside that
+band, which is the point: they were unfalsifiable rather than wrong.
+
+Refusal is cheap for a structural reason, not a tuned one: an unanswerable
+query touches only the posting lists of its own distinctive words, and never
+reaches ranking at all.
 
 **Scale**, synthetic 10,000-unit repository (500 files):
 
@@ -350,8 +395,11 @@ touches only the posting lists of its own distinctive words, never the corpus.
 | with a usable signature | **91 / 91** |
 | units invented that do not exist | **0** |
 
-Directional local measurements, not service levels. Reproduce with
-`python benchmarks/repo_queries.py` and `python benchmarks/large_repo.py`.
+Directional local measurements, not service levels — but every one of them is
+now a command rather than a memory, which two of them were not before. Each
+prints the corpus fingerprint beside its score; quote both or neither.
+[`benchmarks/README.md`](benchmarks/README.md) lists the six scripts and what
+each is for, and the corpus one of them grades is now carried here too.
 
 ## 7 · `rag-your-code search` vs a Grep loop
 
@@ -361,45 +409,74 @@ many hit. That is what this reproduces — same corpus, same questions, same
 ruler, scored at **file** granularity so Grep is not penalised for lacking
 declaration spans.
 
-**On a repository nobody has described, Grep wins.** That is the measured
-result and it is not softened here.
+**Which side wins on an undescribed repository depends on the repository.**
+Through 1.3.0 this section said flatly that Grep wins there, because the one
+undescribed repository ever measured was a hook-heavy tool whose questions were
+answerable by matching identifiers. Swapping the subject for a public web
+framework reversed it. The honest claim is narrower than either table alone:
 
-| foreign repository · 35 questions · 1,267 units · no descriptions | Grep loop | rag-your-code |
+| Flask 3.1.3 · 35 questions · 1,572 units `5fd51169eacc` · no descriptions | Grep loop | rag-your-code |
 |---|---|---|
-| right file first | **34.3%** | 31.4% |
-| right file in top 3 | **60.0%** | 48.6% |
-| lines matched across the repo, all questions | 33,213 | — |
-| characters returned, all questions | — | **163,294** |
-| questions it answers | 35 | 28 |
+| right file first | 22.9% | **37.1%** |
+| right file in top 3 | 45.7% | **57.1%** |
+| lines it hands back, all questions | 17,641 | — |
+| characters returned, all questions | 1,415,656 | **249,720** |
+| questions it answers | 30 | 30 |
 
 **Once the vocabulary exists, it is not close.**
 
-| this repository · 70 questions · 569 units · 304 described | Grep loop | rag-your-code |
+| this repository · 70 questions · 581 units `978a1d48a82a` · 304 described | Grep loop | rag-your-code |
 |---|---|---|
-| right file first | 25.7% | **58.6%** |
-| right file in top 3 | 64.3% | **75.7%** |
-| lines matched across the repo, all questions | 40,150 | — |
-| characters returned, all questions | — | **277,327** |
-| questions it answers | 70 | 60 |
+| right file first | 22.9% | **58.6%** |
+| right file in top 3 | 54.3% | **75.7%** |
+| lines it hands back, all questions | 11,833 | — |
+| characters returned, all questions | 1,122,902 | **626,022** |
+| questions it answers | **61** | 60 |
 
 Those two tables are the whole argument of section 3.3, measured against a real
 baseline instead of asserted. A cold index retrieves against a sentence the
-parser generated from identifiers the author already chose — so it is competing
-with Grep using Grep's own information, and losing, because Grep does not have
-to guess which of the matching files is the definition. Descriptions put words
-in the index that the source never contained, and first-place accuracy goes
-from below Grep's to **more than double** it.
-
-Three qualifications, because the table would otherwise flatter both sides:
+parser generated from identifiers the author already chose, plus whatever
+docstrings the author wrote — so how it fares against Grep is decided by how
+much prose the repository already contains. Flask has a written docstring on
+most public methods, and the cold index beats Grep there without a single
+description being added. On the previous subject, a tool with terse comments
+and long identifiers, the same cold index lost to Grep by the same margin.
+
+What does not depend on the subject is what descriptions buy: on this
+repository first-place accuracy goes to **more than double** Grep's, and the
+payload comes back ranked, spanned, and roughly half the size.
+
+**Both tables come from `python -m benchmarks.grep_baseline`**, which is what
+changed in 1.3.0. Until then this section — the strongest claim the project
+makes — was published from a script that had never been committed, so nothing
+here could be checked and the word "Grep loop" had no precise meaning. The
+committed version defines it: take the query's words, drop the ones the corpus
+itself shows are everywhere, run one substring search per remaining word over
+exactly the files the index was built from, rank each file by how many distinct
+words hit it, break ties on path. Reconstructing it reproduced this side's
+figures exactly and moved Grep's, which is the expected shape — the ranked arm
+was always a call into shipped code, and the baseline never was.
+
+Four qualifications, because the table would otherwise flatter both sides:
 
 - **Scored at file granularity**, which understates this side. A Grep hit is a
   file; a hit here is a declaration with an exact span, a score, and the words
   it matched on. The agent that reads the result opens 40 lines, not a file.
-- **Grep answers everything.** It never declines, which is why it hands back
-  33,213 matching lines for 35 questions — about 950 lines per question, no
-  ranking, no spans, no indication which match is the definition. This returns
-  roughly 5,800 characters per question, ranked. Seven of 35 and ten of 70
-  questions come back empty here instead, with a reason.
+- **Dropping the corpus-common words is generous to Grep**, and it is what
+  makes the baseline a fair one rather than a straw man: an agent that greps
+  `the` gets every file back in no order. It is also why Grep declines nine of
+  the seventy questions here — those had no word left that this corpus does not
+  use everywhere.
+- **Payload is counted in characters on both sides.** On this repository Grep
+  hands back 18,400 characters per question it answers, unranked and without
+  spans, against 10,400 here, ranked and capped by `search.max_chars` — a
+  factor of 1.8. On Flask it is a factor of 5.7 — 47,200 characters against
+  8,300 — because a framework repeats its own vocabulary across many files and
+  Grep has no way to rank what it finds. Both sides decline the same five of
+  those 35 — and they are the five Chinese ones, all of them. A Chinese word is
+  not a substring of English source and it is not a token in an index built
+  from English source, so on a repository written in one language the cold
+  cross-language case is not this tool's failure but the corpus's.
 - **Grep wins outright when you know the string.** `grep -rn "COMMON_TERM"` is
   exact, instant and complete, and nothing here replaces it.
 
@@ -464,19 +541,23 @@ The extra is optional by construction: `dependencies = []` is what a default
 install gets, the import happens inside the constructor, and a test asserts the
 default provider imports none of it.
 
-**Measured, on the same four rulers.** This is the first release with a real
-model behind these numbers — 0.8.0 shipped the seam and said plainly that its
-benefit was unmeasured.
+**Measured on the same four rulers, both arms against one corpus.** 1.1.0
+published this comparison and read it as a win. Its largest gain was on the
+foreign ruler, whose two arms turned out to have been taken against two
+different states of a repository being edited while the script ran. Repeated
+against a pinned corpus:
 
-| ruler | signed hash (default) | MiniLM, local |
-|---|---|---|
-| **A** foreign, cold | 0.229 / 0.400 / 0.300 | **0.286 / 0.457 / 0.357** |
-| **B** own, cold | 0.314 / 0.471 / 0.383 | **0.329 / 0.486 / 0.400** |
-| **C** own, described | 0.443 / 0.614 / 0.507 | 0.443 / **0.671 / 0.540** |
-| **D** silence, own / foreign | 0.967 / 0.933 | 0.967 / 0.933 |
+| ruler | corpus | signed hash (default) | MiniLM, local |
+|---|---|---|---|
+| **A** foreign, cold | 1,572 `5fd51169eacc` | **0.200 / 0.286 / 0.238** | 0.171 / 0.257 / 0.214 |
+| **B** own, cold | 581 `8e1e71942c1c` | 0.314 / 0.471 / 0.383 | 0.314 / 0.471 / 0.383 |
+| **C** own, described | 581 `978a1d48a82a` | **0.443 / 0.614 / 0.507** | 0.429 / 0.600 / 0.500 |
+| **D** silence, own / foreign | as above | 0.967 / 0.833 | 0.967 / 0.833 |
 
-Better on every positive ruler, with refusal unchanged. The pairs the hash
-scores exactly zero:
+**Worse or identical on every ruler.** It is shipped anyway, as an extra nobody
+has to install, because it does one thing the hash cannot do at all and these
+rulers cannot see: reach a unit that shares no word with the question. The
+pairs the hash scores exactly zero:
 
 | pair | signed hash | MiniLM |
 |---|---|---|
@@ -494,6 +575,10 @@ a threshold on a score and the distributions overlap (0.469 vs 0.418 median),
 and a scale-free standout metric took ruler B from 0.329 to 0.186 for two
 thirds of the silence. Applying the lexical bars costs ruler A nothing.
 
+If you install it expecting the hit rates above to move, they will not. Install
+it for the cross-language and paraphrase cases in the table above, which is
+where the difference between the two columns actually lives.
+
 **A hosted endpoint** is the third option, and the only one that sends your
 source anywhere:
 

{rag_your_code-1.2.1 → rag_your_code-1.4.0}/src/ragyourcode/__init__.py

@@ -3,4 +3,4 @@
 from .models import CodeUnit, SearchResult
 
 __all__ = ["CodeUnit", "SearchResult"]
-__version__ = "1.2.1"
+__version__ = "1.4.0"

{rag_your_code-1.2.1 → rag_your_code-1.4.0}/src/ragyourcode/config.py

@@ -201,10 +201,15 @@ SETTINGS: tuple[Setting, ...] = (
     ),
     # How much of a question has to reach the index before an answer counts as
     # evidence rather than as a guess. Measured, not chosen: across two
-    # repositories, two languages and 126 questions, 0.40 is the largest value
-    # at which every question that was being answered correctly still is, and
-    # it silences 47% of questions whose answer is not in the repository at
-    # all. It is a ratio inside the query, so unlike a score threshold it does
+    # repositories, two languages and every question set under `benchmarks/`,
+    # 0.40 is the largest value at which every question that was being answered
+    # correctly still is. On its own it silences three fifths of the questions
+    # whose answer is not in the repository at all; the rest is what
+    # `search.min_concentration` below adds. The command is the claim --
+    # `repo_queries --questions benchmarks/absent_queries.json
+    # --min-concentration 0` -- because a count typed into a comment is a figure
+    # nothing checks, and both numbers this sentence used to carry had rotted.
+    # It is a ratio inside the query, so unlike a score threshold it does
     # not move when the corpus or the scale of the ranking does -- the defect
     # that made `confidence_threshold = 0.8` stop meaning anything.
     Setting(
@@ -221,8 +226,11 @@ SETTINGS: tuple[Setting, ...] = (
     # unrelated declarations -- four of six words found in four places with
     # nothing to do with one another or with what was asked. Measured across
     # four rulers, requiring a quarter of a query's rarity to land inside one
-    # unit leaves the two rulers over undescribed code unchanged and roughly
-    # halves the questions that get an answer they should not have. Rarity-
+    # unit leaves the two rulers over undescribed code unchanged and removes
+    # most of what the coverage bar alone still answers; the ablation table
+    # carries the numbers, in docs/ROADMAP.md, rather than this line, which
+    # said "roughly halves" while that table said an order of magnitude.
+    # Rarity-
     # weighted rather than counted, because a unit holding two ordinary words is
     # not better evidence than one holding the rare word the question is about.
     Setting(

{rag_your_code-1.2.1 → rag_your_code-1.4.0}/src/ragyourcode/search.py

@@ -119,7 +119,7 @@ class Evidence:
 
     Ranking answers "which of these is best". It cannot answer "is any of this
     an answer", and reading the first as the second is what let a repository
-    reply to all 32 questions in `benchmarks/absent_queries.json` -- every one
+    reply to every question in `benchmarks/absent_queries.json` -- each one
     about a subject neither repository implements. `where are CUDA kernels
     dispatched to the device` came back with a test about word counting, on the
     evidence of `are`, `the`, `to` and `where`. That is not a Chinese problem

{rag_your_code-1.2.1 → rag_your_code-1.4.0}/tests/test_absent_queries.py

@@ -53,23 +53,56 @@ def test_the_absent_ruler_is_well_formed():
     assert questions["why"].strip() and questions["caveat"].strip()
 
 
-def test_no_subject_of_an_absent_question_exists_in_this_repository(units):
-    """The absence claim, re-derived rather than trusted.
-
-    This is the assertion the ruler's own caveat promises, and it is the one
-    that fails first when the repository grows into a subject the ruler
-    assumed it would never contain.
+def _intruders(units) -> list[tuple[str, str, int]]:
+    """Which absent question's own vocabulary reaches a corpus that must not
+    contain it.
     """
     index = build_search_index(units)
-    intruders = [
+    return [
         (entry["id"], term, len(index.postings[term]))
         for entry in load_questions(ABSENT_PATH)["queries"]
         for term in entry["subject"]
         if index.postings.get(term)
     ]
-    assert not intruders, (
+
+
+def test_no_subject_of_an_absent_question_exists_in_this_repository(units):
+    """The absence claim, re-derived rather than trusted.
+
+    This is the assertion the ruler's own caveat promises, and it is the one
+    that fails first when the repository grows into a subject the ruler
+    assumed it would never contain.
+    """
+    assert not _intruders(units), (
         "this repository now contains the vocabulary of a question the ruler calls unanswerable; "
-        f"retire or rewrite those questions rather than letting them score as misses: {intruders}"
+        f"retire or rewrite those questions rather than letting them score as misses: {_intruders(units)}"
+    )
+
+
+def test_no_subject_of_an_absent_question_exists_in_the_vendored_corpus():
+    """The other half of the same claim, which used to be unenforceable.
+
+    This ruler asserts its questions are unanswerable in *both* graded
+    repositories. Until the foreign one was vendored that half could only be
+    asserted, because the subject was somebody else's checkout and a re-run
+    here could not see what it had become. It is now a directory in this
+    repository at a pinned tag, so the claim is a test.
+
+    It has already earned its place. One question's subject list carried a
+    generic English verb where a specific term belonged, and the corpus below
+    uses that verb in a comment about signing keys -- so the question was one
+    ordinary word away from being answerable by accident. The verb was
+    replaced with a term that names the thing.
+
+    Naming the subject here would put it in the index and break the very
+    claim this asserts, which is why the paragraph above is written around it.
+    That has now happened five times; CONTRIBUTING.md keeps the tally.
+    """
+    corpus = ROOT / "benchmarks" / "corpus" / "flask"
+    assert corpus.is_dir(), f"the vendored corpus is missing from {corpus}"
+    assert not _intruders(build_units(corpus)), (
+        "the vendored corpus contains the vocabulary of a question the ruler calls unanswerable; "
+        f"rewrite the question rather than letting it score as a miss: {_intruders(build_units(corpus))}"
     )
 
 

{rag_your_code-1.2.1 → rag_your_code-1.4.0}/tests/test_evidence.py

@@ -1,9 +1,10 @@
 """Retrieval must be able to say it has no answer.
 
 Ranking always produces a least-bad unit and returns it with a score and a
-rank, which read exactly like an answer. Graded against 32 questions about
-subjects neither this repository nor `benchmarks/cold_queries.json`'s
-repository implements, every single one came back answered -- `where are CUDA
+rank, which read exactly like an answer. Graded against every question in
+`benchmarks/absent_queries.json` -- subjects neither this repository nor
+`benchmarks/cold_queries.json`'s repository implements -- every single one came
+back answered, on both repositories. `where are CUDA
 kernels dispatched to the device` on the evidence of `are`, `the`, `to` and
 `where`. These assert the second question retrieval now asks: not which unit
 ranks highest, but whether any of this is evidence at all.

{rag_your_code-1.2.1 → rag_your_code-1.4.0}/tests/test_metadata.py

@@ -321,3 +321,35 @@ def test_every_documented_provider_block_actually_configures_that_provider(tmp_p
         # arrangement exists to prevent.
         assert "sk-" not in settings
     assert {"sentence-transformers", "openai-compatible"} <= seen, f"undocumented providers; README shows {sorted(seen)}"
+
+
+# --- a measurement nobody can re-run is a claim, not a measurement ----------
+
+BENCHMARKS = tuple(
+    sorted(
+        path.name
+        for path in (ROOT / "benchmarks").glob("*.py")
+        if path.name != "__init__.py"
+    )
+)
+
+
+def test_every_benchmark_script_is_listed_in_its_own_index():
+    """The index of the rulers is discovered, not maintained by hand.
+
+    1.3.0 found two figures in the README -- query latency, and the entire Grep
+    head-to-head -- produced by scripts that had never been committed. Nobody
+    could re-derive either, and the second one meant the phrase "a Grep loop"
+    had no definition a reader could argue with.
+
+    Both directions matter. A script absent from the index is one nobody knows
+    to run; a command in the index naming a script that does not exist is the
+    install-line defect this repository shipped twice. Discovery by glob is
+    what keeps the seventh script from being the one nothing checks.
+    """
+    index = (ROOT / "benchmarks" / "README.md").read_text(encoding="utf-8")
+    assert BENCHMARKS, "benchmarks/ holds no scripts; this guard would pass vacuously"
+    for name in BENCHMARKS:
+        assert f"benchmarks.{Path(name).stem}" in index, f"{name} is in benchmarks/ and not in its README"
+    for module in re.findall(r"benchmarks\.([a-z_]+)", index):
+        assert (ROOT / "benchmarks" / f"{module}.py").is_file(), f"the README runs benchmarks.{module}, which does not exist"