rag-your-code 1.2.0__tar.gz → 1.3.0__tar.gz

{rag_your_code-1.2.0/src/rag_your_code.egg-info → rag_your_code-1.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rag-your-code
-Version: 1.2.0
+Version: 1.3.0
 Summary: A local, explainable RAG index for codebases and coding agents
 Author: rag-your-code contributors
 License-Expression: MIT
@@ -293,10 +293,18 @@ repository had grown by ninety units.
 
 | ruler | what it represents | n | hit@1 | hit@3 | MRR |
 |---|---|---|---|---|---|
-| **A** foreign repo, no descriptions | what a first-time user gets | 35 | 0.229 | 0.400 | 0.300 |
+| **A** foreign repo, no descriptions | what a first-time user gets | 35 | 0.229 | 0.371 | 0.286 |
 | **B** this repo, generated descriptions only | a cold index of familiar code | 70 | 0.314 | 0.471 | 0.383 |
 | **C** this repo, agent-written descriptions | the warmest case supported | 70 | 0.443 | 0.614 | 0.507 |
 
+The corpora, without which none of the above is reproducible — **A**
+cc-enforcer, 1,345 units, `471b78f9f806`; **B** 579 units, `dca2a4656659`;
+**C** 579 units, `0a2f050dbafd`. Ruler A's repository is one this project does
+not control. Between the previous release and this table it grew by 78 units
+and renamed a declaration two questions pointed at, which stopped the ruler
+running at all until they were repointed — hit@3 0.400 → 0.371 and MRR 0.300 →
+0.286 are that, not a change in retrieval.
+
 **Refusal — the fourth ruler, 30 questions with no answer anywhere**
 
 | | this repo | foreign repo |
@@ -310,25 +318,39 @@ repository had grown by ninety units.
 
 | gate | A hit@1/3/MRR | B hit@1/3/MRR | C hit@1/3/MRR | silence own / foreign |
 |---|---|---|---|---|
-| neither (pre-1.0.0) | 0.229/0.400/0.300 | 0.314/0.486/0.391 | 0.471/0.686/0.552 | 0.000 / 0.000 |
-| coverage only (1.0.0) | 0.229/0.400/0.300 | 0.314/0.471/0.383 | 0.471/0.671/0.548 | 0.700 / 0.767 |
-| **both (1.1.0)** | **0.229/0.400/0.300** | **0.314/0.471/0.383** | 0.443/0.614/0.507 | **0.967 / 0.933** |
+| neither (pre-1.0.0) | 0.229/0.371/0.286 | 0.314/0.486/0.391 | 0.486/0.700/0.567 | 0.000 / 0.000 |
+| coverage only (1.0.0) | 0.229/0.371/0.286 | 0.314/0.471/0.383 | 0.486/0.686/0.562 | 0.600 / 0.767 |
+| **both (1.1.0)** | **0.229/0.371/0.286** | **0.314/0.471/0.383** | 0.443/0.614/0.507 | **0.967 / 0.933** |
 
-Rulers A and B are **identical to three decimals**. The entire cost is four
-questions of seventy on the warmest ruler. On these four rulers concentration
+Rulers A and B are **identical to three decimals**. The entire cost is three
+questions of seventy at hit@1 on the warmest ruler, and six at hit@3. On these
+four rulers concentration
 subsumes coverage — stated plainly because it is true; coverage is kept because
 it answers a different question and names a different diagnosis.
 
-**Latency** — warm corpus, 557 units, 420 samples after warm-up:
-
-| | |
-|---|---|
-| query, median | **0.83 ms** |
-| query, p95 | 1.68 ms |
-| refusing an unanswerable query | **0.03 ms** |
+**Latency** — warm corpus, 579 units `0a2f050dbafd`, five consecutive
+invocations of `python -m benchmarks.query_latency --repeats 10` (420 samples
+each):
 
-Refusal is cheaper than answering by a factor of forty: an unanswerable query
-touches only the posting lists of its own distinctive words, never the corpus.
+| | | across the five |
+|---|---|---|
+| query, median | **0.65 ms** | 0.62 – 0.92 |
+| query, p95 | 1.13 ms | 1.09 – 1.48 |
+| refusing an unanswerable query | **0.022 ms** | 0.019 – 0.024 |
+| refusal cheaper than answering by | **~35×** | 31 – 39 |
+
+Two significant figures and a spread, because that is the precision the
+measurement has. A second set of five taken earlier the same hour, while the
+machine was busy, put the median at 1.0 ms and p95 at 3.0 ms; the gap between
+the two sets is the machine, and it is wider than any change the code has ever
+made to this number. Earlier releases published `0.83 ms / p95 1.68 ms` to
+three figures from a script that was never committed, on a corpus that no
+longer exists — both values sit inside today's range, which is the point: they
+were unfalsifiable rather than wrong.
+
+Refusal is cheap for a structural reason, not a tuned one: an unanswerable
+query touches only the posting lists of its own distinctive words, and never
+reaches ranking at all.
 
 **Scale**, synthetic 10,000-unit repository (500 files):
 
@@ -350,8 +372,11 @@ touches only the posting lists of its own distinctive words, never the corpus.
 | with a usable signature | **91 / 91** |
 | units invented that do not exist | **0** |
 
-Directional local measurements, not service levels. Reproduce with
-`python benchmarks/repo_queries.py` and `python benchmarks/large_repo.py`.
+Directional local measurements, not service levels — but every one of them is
+now a command rather than a memory, which two of them were not before. Each
+prints the corpus fingerprint beside its score; quote both or neither.
+[`benchmarks/README.md`](benchmarks/README.md) lists the five scripts and what
+each is for.
 
 ## 7 · `rag-your-code search` vs a Grep loop
 
@@ -364,23 +389,23 @@ declaration spans.
 **On a repository nobody has described, Grep wins.** That is the measured
 result and it is not softened here.
 
-| foreign repository · 35 questions · 1,267 units · no descriptions | Grep loop | rag-your-code |
+| foreign repository · 35 questions · 1,345 units `471b78f9f806` · no descriptions | Grep loop | rag-your-code |
 |---|---|---|
-| right file first | **34.3%** | 31.4% |
-| right file in top 3 | **60.0%** | 48.6% |
-| lines matched across the repo, all questions | 33,213 | — |
-| characters returned, all questions | — | **163,294** |
-| questions it answers | 35 | 28 |
+| right file first | **40.0%** | 28.6% |
+| right file in top 3 | **62.9%** | 48.6% |
+| lines it hands back, all questions | 8,937 | — |
+| characters returned, all questions | 841,484 | **282,102** |
+| questions it answers | **35** | 28 |
 
 **Once the vocabulary exists, it is not close.**
 
-| this repository · 70 questions · 569 units · 304 described | Grep loop | rag-your-code |
+| this repository · 70 questions · 579 units `0a2f050dbafd` · 304 described | Grep loop | rag-your-code |
 |---|---|---|
-| right file first | 25.7% | **58.6%** |
-| right file in top 3 | 64.3% | **75.7%** |
-| lines matched across the repo, all questions | 40,150 | — |
-| characters returned, all questions | — | **277,327** |
-| questions it answers | 70 | 60 |
+| right file first | 22.9% | **58.6%** |
+| right file in top 3 | 55.7% | **75.7%** |
+| lines it hands back, all questions | 11,636 | — |
+| characters returned, all questions | 1,101,618 | **621,837** |
+| questions it answers | **61** | 60 |
 
 Those two tables are the whole argument of section 3.3, measured against a real
 baseline instead of asserted. A cold index retrieves against a sentence the
@@ -390,15 +415,32 @@ to guess which of the matching files is the definition. Descriptions put words
 in the index that the source never contained, and first-place accuracy goes
 from below Grep's to **more than double** it.
 
-Three qualifications, because the table would otherwise flatter both sides:
+**Both tables come from `python -m benchmarks.grep_baseline`**, which is what
+changed in 1.3.0. Until then this section — the strongest claim the project
+makes — was published from a script that had never been committed, so nothing
+here could be checked and the word "Grep loop" had no precise meaning. The
+committed version defines it: take the query's words, drop the ones the corpus
+itself shows are everywhere, run one substring search per remaining word over
+exactly the files the index was built from, rank each file by how many distinct
+words hit it, break ties on path. Reconstructing it reproduced this side's
+figures exactly and moved Grep's, which is the expected shape — the ranked arm
+was always a call into shipped code, and the baseline never was.
+
+Four qualifications, because the table would otherwise flatter both sides:
 
 - **Scored at file granularity**, which understates this side. A Grep hit is a
   file; a hit here is a declaration with an exact span, a score, and the words
   it matched on. The agent that reads the result opens 40 lines, not a file.
-- **Grep answers everything.** It never declines, which is why it hands back
-  33,213 matching lines for 35 questions — about 950 lines per question, no
-  ranking, no spans, no indication which match is the definition. This returns
-  roughly 5,800 characters per question, ranked. Seven of 35 and ten of 70
+- **Dropping the corpus-common words is generous to Grep**, and it is what
+  makes the baseline a fair one rather than a straw man: an agent that greps
+  `the` gets every file back in no order. It is also why Grep declines nine of
+  the seventy questions here — those had no word left that this corpus does not
+  use everywhere.
+- **Payload is counted in characters on both sides.** Grep hands back 18,000
+  characters per question it answers, unranked and without spans; this returns
+  10,400, ranked, capped by `search.max_chars`. That is a factor of 1.7, not
+  the order of magnitude an earlier version of this table implied by counting
+  one side in lines and the other in characters. Seven of 35 and ten of 70
   questions come back empty here instead, with a reason.
 - **Grep wins outright when you know the string.** `grep -rn "COMMON_TERM"` is
   exact, instant and complete, and nothing here replaces it.
@@ -651,6 +693,16 @@ need a dependency, and the policy for those is settled: they follow the
 embedding provider's pattern — optional, user-selected, never in the default
 install. Full reasoning in [docs/ROADMAP.md](docs/ROADMAP.md).
 
+**Whether the skill fires unprompted is not measured**, and until 1.2.1 this
+project claimed no command could measure it. That was wrong: `claude plugin
+eval` grades exactly this, with `tool_used: Skill` as a plugin-fired indicator
+and a no-plugin baseline arm. It is unmeasured because the command is in early
+access on the account here and its case schema is undocumented, so a suite
+written from `--help` fragments could not be run even once to see whether it
+loads — and a suite that silently fails to load reads as a gate while checking
+nothing. Since 1.2.0 the four commands give an entry path that does not depend
+on it.
+
 ## 12 · Development
 
 ```bash

{rag_your_code-1.2.0 → rag_your_code-1.3.0}/README.md

@@ -264,10 +264,18 @@ repository had grown by ninety units.
 
 | ruler | what it represents | n | hit@1 | hit@3 | MRR |
 |---|---|---|---|---|---|
-| **A** foreign repo, no descriptions | what a first-time user gets | 35 | 0.229 | 0.400 | 0.300 |
+| **A** foreign repo, no descriptions | what a first-time user gets | 35 | 0.229 | 0.371 | 0.286 |
 | **B** this repo, generated descriptions only | a cold index of familiar code | 70 | 0.314 | 0.471 | 0.383 |
 | **C** this repo, agent-written descriptions | the warmest case supported | 70 | 0.443 | 0.614 | 0.507 |
 
+The corpora, without which none of the above is reproducible — **A**
+cc-enforcer, 1,345 units, `471b78f9f806`; **B** 579 units, `dca2a4656659`;
+**C** 579 units, `0a2f050dbafd`. Ruler A's repository is one this project does
+not control. Between the previous release and this table it grew by 78 units
+and renamed a declaration two questions pointed at, which stopped the ruler
+running at all until they were repointed — hit@3 0.400 → 0.371 and MRR 0.300 →
+0.286 are that, not a change in retrieval.
+
 **Refusal — the fourth ruler, 30 questions with no answer anywhere**
 
 | | this repo | foreign repo |
@@ -281,25 +289,39 @@ repository had grown by ninety units.
 
 | gate | A hit@1/3/MRR | B hit@1/3/MRR | C hit@1/3/MRR | silence own / foreign |
 |---|---|---|---|---|
-| neither (pre-1.0.0) | 0.229/0.400/0.300 | 0.314/0.486/0.391 | 0.471/0.686/0.552 | 0.000 / 0.000 |
-| coverage only (1.0.0) | 0.229/0.400/0.300 | 0.314/0.471/0.383 | 0.471/0.671/0.548 | 0.700 / 0.767 |
-| **both (1.1.0)** | **0.229/0.400/0.300** | **0.314/0.471/0.383** | 0.443/0.614/0.507 | **0.967 / 0.933** |
+| neither (pre-1.0.0) | 0.229/0.371/0.286 | 0.314/0.486/0.391 | 0.486/0.700/0.567 | 0.000 / 0.000 |
+| coverage only (1.0.0) | 0.229/0.371/0.286 | 0.314/0.471/0.383 | 0.486/0.686/0.562 | 0.600 / 0.767 |
+| **both (1.1.0)** | **0.229/0.371/0.286** | **0.314/0.471/0.383** | 0.443/0.614/0.507 | **0.967 / 0.933** |
 
-Rulers A and B are **identical to three decimals**. The entire cost is four
-questions of seventy on the warmest ruler. On these four rulers concentration
+Rulers A and B are **identical to three decimals**. The entire cost is three
+questions of seventy at hit@1 on the warmest ruler, and six at hit@3. On these
+four rulers concentration
 subsumes coverage — stated plainly because it is true; coverage is kept because
 it answers a different question and names a different diagnosis.
 
-**Latency** — warm corpus, 557 units, 420 samples after warm-up:
-
-| | |
-|---|---|
-| query, median | **0.83 ms** |
-| query, p95 | 1.68 ms |
-| refusing an unanswerable query | **0.03 ms** |
+**Latency** — warm corpus, 579 units `0a2f050dbafd`, five consecutive
+invocations of `python -m benchmarks.query_latency --repeats 10` (420 samples
+each):
 
-Refusal is cheaper than answering by a factor of forty: an unanswerable query
-touches only the posting lists of its own distinctive words, never the corpus.
+| | | across the five |
+|---|---|---|
+| query, median | **0.65 ms** | 0.62 – 0.92 |
+| query, p95 | 1.13 ms | 1.09 – 1.48 |
+| refusing an unanswerable query | **0.022 ms** | 0.019 – 0.024 |
+| refusal cheaper than answering by | **~35×** | 31 – 39 |
+
+Two significant figures and a spread, because that is the precision the
+measurement has. A second set of five taken earlier the same hour, while the
+machine was busy, put the median at 1.0 ms and p95 at 3.0 ms; the gap between
+the two sets is the machine, and it is wider than any change the code has ever
+made to this number. Earlier releases published `0.83 ms / p95 1.68 ms` to
+three figures from a script that was never committed, on a corpus that no
+longer exists — both values sit inside today's range, which is the point: they
+were unfalsifiable rather than wrong.
+
+Refusal is cheap for a structural reason, not a tuned one: an unanswerable
+query touches only the posting lists of its own distinctive words, and never
+reaches ranking at all.
 
 **Scale**, synthetic 10,000-unit repository (500 files):
 
@@ -321,8 +343,11 @@ touches only the posting lists of its own distinctive words, never the corpus.
 | with a usable signature | **91 / 91** |
 | units invented that do not exist | **0** |
 
-Directional local measurements, not service levels. Reproduce with
-`python benchmarks/repo_queries.py` and `python benchmarks/large_repo.py`.
+Directional local measurements, not service levels — but every one of them is
+now a command rather than a memory, which two of them were not before. Each
+prints the corpus fingerprint beside its score; quote both or neither.
+[`benchmarks/README.md`](benchmarks/README.md) lists the five scripts and what
+each is for.
 
 ## 7 · `rag-your-code search` vs a Grep loop
 
@@ -335,23 +360,23 @@ declaration spans.
 **On a repository nobody has described, Grep wins.** That is the measured
 result and it is not softened here.
 
-| foreign repository · 35 questions · 1,267 units · no descriptions | Grep loop | rag-your-code |
+| foreign repository · 35 questions · 1,345 units `471b78f9f806` · no descriptions | Grep loop | rag-your-code |
 |---|---|---|
-| right file first | **34.3%** | 31.4% |
-| right file in top 3 | **60.0%** | 48.6% |
-| lines matched across the repo, all questions | 33,213 | — |
-| characters returned, all questions | — | **163,294** |
-| questions it answers | 35 | 28 |
+| right file first | **40.0%** | 28.6% |
+| right file in top 3 | **62.9%** | 48.6% |
+| lines it hands back, all questions | 8,937 | — |
+| characters returned, all questions | 841,484 | **282,102** |
+| questions it answers | **35** | 28 |
 
 **Once the vocabulary exists, it is not close.**
 
-| this repository · 70 questions · 569 units · 304 described | Grep loop | rag-your-code |
+| this repository · 70 questions · 579 units `0a2f050dbafd` · 304 described | Grep loop | rag-your-code |
 |---|---|---|
-| right file first | 25.7% | **58.6%** |
-| right file in top 3 | 64.3% | **75.7%** |
-| lines matched across the repo, all questions | 40,150 | — |
-| characters returned, all questions | — | **277,327** |
-| questions it answers | 70 | 60 |
+| right file first | 22.9% | **58.6%** |
+| right file in top 3 | 55.7% | **75.7%** |
+| lines it hands back, all questions | 11,636 | — |
+| characters returned, all questions | 1,101,618 | **621,837** |
+| questions it answers | **61** | 60 |
 
 Those two tables are the whole argument of section 3.3, measured against a real
 baseline instead of asserted. A cold index retrieves against a sentence the
@@ -361,15 +386,32 @@ to guess which of the matching files is the definition. Descriptions put words
 in the index that the source never contained, and first-place accuracy goes
 from below Grep's to **more than double** it.
 
-Three qualifications, because the table would otherwise flatter both sides:
+**Both tables come from `python -m benchmarks.grep_baseline`**, which is what
+changed in 1.3.0. Until then this section — the strongest claim the project
+makes — was published from a script that had never been committed, so nothing
+here could be checked and the word "Grep loop" had no precise meaning. The
+committed version defines it: take the query's words, drop the ones the corpus
+itself shows are everywhere, run one substring search per remaining word over
+exactly the files the index was built from, rank each file by how many distinct
+words hit it, break ties on path. Reconstructing it reproduced this side's
+figures exactly and moved Grep's, which is the expected shape — the ranked arm
+was always a call into shipped code, and the baseline never was.
+
+Four qualifications, because the table would otherwise flatter both sides:
 
 - **Scored at file granularity**, which understates this side. A Grep hit is a
   file; a hit here is a declaration with an exact span, a score, and the words
   it matched on. The agent that reads the result opens 40 lines, not a file.
-- **Grep answers everything.** It never declines, which is why it hands back
-  33,213 matching lines for 35 questions — about 950 lines per question, no
-  ranking, no spans, no indication which match is the definition. This returns
-  roughly 5,800 characters per question, ranked. Seven of 35 and ten of 70
+- **Dropping the corpus-common words is generous to Grep**, and it is what
+  makes the baseline a fair one rather than a straw man: an agent that greps
+  `the` gets every file back in no order. It is also why Grep declines nine of
+  the seventy questions here — those had no word left that this corpus does not
+  use everywhere.
+- **Payload is counted in characters on both sides.** Grep hands back 18,000
+  characters per question it answers, unranked and without spans; this returns
+  10,400, ranked, capped by `search.max_chars`. That is a factor of 1.7, not
+  the order of magnitude an earlier version of this table implied by counting
+  one side in lines and the other in characters. Seven of 35 and ten of 70
   questions come back empty here instead, with a reason.
 - **Grep wins outright when you know the string.** `grep -rn "COMMON_TERM"` is
   exact, instant and complete, and nothing here replaces it.
@@ -622,6 +664,16 @@ need a dependency, and the policy for those is settled: they follow the
 embedding provider's pattern — optional, user-selected, never in the default
 install. Full reasoning in [docs/ROADMAP.md](docs/ROADMAP.md).
 
+**Whether the skill fires unprompted is not measured**, and until 1.2.1 this
+project claimed no command could measure it. That was wrong: `claude plugin
+eval` grades exactly this, with `tool_used: Skill` as a plugin-fired indicator
+and a no-plugin baseline arm. It is unmeasured because the command is in early
+access on the account here and its case schema is undocumented, so a suite
+written from `--help` fragments could not be run even once to see whether it
+loads — and a suite that silently fails to load reads as a gate while checking
+nothing. Since 1.2.0 the four commands give an entry path that does not depend
+on it.
+
 ## 12 · Development
 
 ```bash

{rag_your_code-1.2.0 → rag_your_code-1.3.0}/pyproject.toml

@@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "rag-your-code"
-version = "1.2.0"
+version = "1.3.0"
 description = "A local, explainable RAG index for codebases and coding agents"
 readme = "README.md"
 requires-python = ">=3.10"

{rag_your_code-1.2.0 → rag_your_code-1.3.0/src/rag_your_code.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rag-your-code
-Version: 1.2.0
+Version: 1.3.0
 Summary: A local, explainable RAG index for codebases and coding agents
 Author: rag-your-code contributors
 License-Expression: MIT
@@ -293,10 +293,18 @@ repository had grown by ninety units.
 
 | ruler | what it represents | n | hit@1 | hit@3 | MRR |
 |---|---|---|---|---|---|
-| **A** foreign repo, no descriptions | what a first-time user gets | 35 | 0.229 | 0.400 | 0.300 |
+| **A** foreign repo, no descriptions | what a first-time user gets | 35 | 0.229 | 0.371 | 0.286 |
 | **B** this repo, generated descriptions only | a cold index of familiar code | 70 | 0.314 | 0.471 | 0.383 |
 | **C** this repo, agent-written descriptions | the warmest case supported | 70 | 0.443 | 0.614 | 0.507 |
 
+The corpora, without which none of the above is reproducible — **A**
+cc-enforcer, 1,345 units, `471b78f9f806`; **B** 579 units, `dca2a4656659`;
+**C** 579 units, `0a2f050dbafd`. Ruler A's repository is one this project does
+not control. Between the previous release and this table it grew by 78 units
+and renamed a declaration two questions pointed at, which stopped the ruler
+running at all until they were repointed — hit@3 0.400 → 0.371 and MRR 0.300 →
+0.286 are that, not a change in retrieval.
+
 **Refusal — the fourth ruler, 30 questions with no answer anywhere**
 
 | | this repo | foreign repo |
@@ -310,25 +318,39 @@ repository had grown by ninety units.
 
 | gate | A hit@1/3/MRR | B hit@1/3/MRR | C hit@1/3/MRR | silence own / foreign |
 |---|---|---|---|---|
-| neither (pre-1.0.0) | 0.229/0.400/0.300 | 0.314/0.486/0.391 | 0.471/0.686/0.552 | 0.000 / 0.000 |
-| coverage only (1.0.0) | 0.229/0.400/0.300 | 0.314/0.471/0.383 | 0.471/0.671/0.548 | 0.700 / 0.767 |
-| **both (1.1.0)** | **0.229/0.400/0.300** | **0.314/0.471/0.383** | 0.443/0.614/0.507 | **0.967 / 0.933** |
+| neither (pre-1.0.0) | 0.229/0.371/0.286 | 0.314/0.486/0.391 | 0.486/0.700/0.567 | 0.000 / 0.000 |
+| coverage only (1.0.0) | 0.229/0.371/0.286 | 0.314/0.471/0.383 | 0.486/0.686/0.562 | 0.600 / 0.767 |
+| **both (1.1.0)** | **0.229/0.371/0.286** | **0.314/0.471/0.383** | 0.443/0.614/0.507 | **0.967 / 0.933** |
 
-Rulers A and B are **identical to three decimals**. The entire cost is four
-questions of seventy on the warmest ruler. On these four rulers concentration
+Rulers A and B are **identical to three decimals**. The entire cost is three
+questions of seventy at hit@1 on the warmest ruler, and six at hit@3. On these
+four rulers concentration
 subsumes coverage — stated plainly because it is true; coverage is kept because
 it answers a different question and names a different diagnosis.
 
-**Latency** — warm corpus, 557 units, 420 samples after warm-up:
-
-| | |
-|---|---|
-| query, median | **0.83 ms** |
-| query, p95 | 1.68 ms |
-| refusing an unanswerable query | **0.03 ms** |
+**Latency** — warm corpus, 579 units `0a2f050dbafd`, five consecutive
+invocations of `python -m benchmarks.query_latency --repeats 10` (420 samples
+each):
 
-Refusal is cheaper than answering by a factor of forty: an unanswerable query
-touches only the posting lists of its own distinctive words, never the corpus.
+| | | across the five |
+|---|---|---|
+| query, median | **0.65 ms** | 0.62 – 0.92 |
+| query, p95 | 1.13 ms | 1.09 – 1.48 |
+| refusing an unanswerable query | **0.022 ms** | 0.019 – 0.024 |
+| refusal cheaper than answering by | **~35×** | 31 – 39 |
+
+Two significant figures and a spread, because that is the precision the
+measurement has. A second set of five taken earlier the same hour, while the
+machine was busy, put the median at 1.0 ms and p95 at 3.0 ms; the gap between
+the two sets is the machine, and it is wider than any change the code has ever
+made to this number. Earlier releases published `0.83 ms / p95 1.68 ms` to
+three figures from a script that was never committed, on a corpus that no
+longer exists — both values sit inside today's range, which is the point: they
+were unfalsifiable rather than wrong.
+
+Refusal is cheap for a structural reason, not a tuned one: an unanswerable
+query touches only the posting lists of its own distinctive words, and never
+reaches ranking at all.
 
 **Scale**, synthetic 10,000-unit repository (500 files):
 
@@ -350,8 +372,11 @@ touches only the posting lists of its own distinctive words, never the corpus.
 | with a usable signature | **91 / 91** |
 | units invented that do not exist | **0** |
 
-Directional local measurements, not service levels. Reproduce with
-`python benchmarks/repo_queries.py` and `python benchmarks/large_repo.py`.
+Directional local measurements, not service levels — but every one of them is
+now a command rather than a memory, which two of them were not before. Each
+prints the corpus fingerprint beside its score; quote both or neither.
+[`benchmarks/README.md`](benchmarks/README.md) lists the five scripts and what
+each is for.
 
 ## 7 · `rag-your-code search` vs a Grep loop
 
@@ -364,23 +389,23 @@ declaration spans.
 **On a repository nobody has described, Grep wins.** That is the measured
 result and it is not softened here.
 
-| foreign repository · 35 questions · 1,267 units · no descriptions | Grep loop | rag-your-code |
+| foreign repository · 35 questions · 1,345 units `471b78f9f806` · no descriptions | Grep loop | rag-your-code |
 |---|---|---|
-| right file first | **34.3%** | 31.4% |
-| right file in top 3 | **60.0%** | 48.6% |
-| lines matched across the repo, all questions | 33,213 | — |
-| characters returned, all questions | — | **163,294** |
-| questions it answers | 35 | 28 |
+| right file first | **40.0%** | 28.6% |
+| right file in top 3 | **62.9%** | 48.6% |
+| lines it hands back, all questions | 8,937 | — |
+| characters returned, all questions | 841,484 | **282,102** |
+| questions it answers | **35** | 28 |
 
 **Once the vocabulary exists, it is not close.**
 
-| this repository · 70 questions · 569 units · 304 described | Grep loop | rag-your-code |
+| this repository · 70 questions · 579 units `0a2f050dbafd` · 304 described | Grep loop | rag-your-code |
 |---|---|---|
-| right file first | 25.7% | **58.6%** |
-| right file in top 3 | 64.3% | **75.7%** |
-| lines matched across the repo, all questions | 40,150 | — |
-| characters returned, all questions | — | **277,327** |
-| questions it answers | 70 | 60 |
+| right file first | 22.9% | **58.6%** |
+| right file in top 3 | 55.7% | **75.7%** |
+| lines it hands back, all questions | 11,636 | — |
+| characters returned, all questions | 1,101,618 | **621,837** |
+| questions it answers | **61** | 60 |
 
 Those two tables are the whole argument of section 3.3, measured against a real
 baseline instead of asserted. A cold index retrieves against a sentence the
@@ -390,15 +415,32 @@ to guess which of the matching files is the definition. Descriptions put words
 in the index that the source never contained, and first-place accuracy goes
 from below Grep's to **more than double** it.
 
-Three qualifications, because the table would otherwise flatter both sides:
+**Both tables come from `python -m benchmarks.grep_baseline`**, which is what
+changed in 1.3.0. Until then this section — the strongest claim the project
+makes — was published from a script that had never been committed, so nothing
+here could be checked and the word "Grep loop" had no precise meaning. The
+committed version defines it: take the query's words, drop the ones the corpus
+itself shows are everywhere, run one substring search per remaining word over
+exactly the files the index was built from, rank each file by how many distinct
+words hit it, break ties on path. Reconstructing it reproduced this side's
+figures exactly and moved Grep's, which is the expected shape — the ranked arm
+was always a call into shipped code, and the baseline never was.
+
+Four qualifications, because the table would otherwise flatter both sides:
 
 - **Scored at file granularity**, which understates this side. A Grep hit is a
   file; a hit here is a declaration with an exact span, a score, and the words
   it matched on. The agent that reads the result opens 40 lines, not a file.
-- **Grep answers everything.** It never declines, which is why it hands back
-  33,213 matching lines for 35 questions — about 950 lines per question, no
-  ranking, no spans, no indication which match is the definition. This returns
-  roughly 5,800 characters per question, ranked. Seven of 35 and ten of 70
+- **Dropping the corpus-common words is generous to Grep**, and it is what
+  makes the baseline a fair one rather than a straw man: an agent that greps
+  `the` gets every file back in no order. It is also why Grep declines nine of
+  the seventy questions here — those had no word left that this corpus does not
+  use everywhere.
+- **Payload is counted in characters on both sides.** Grep hands back 18,000
+  characters per question it answers, unranked and without spans; this returns
+  10,400, ranked, capped by `search.max_chars`. That is a factor of 1.7, not
+  the order of magnitude an earlier version of this table implied by counting
+  one side in lines and the other in characters. Seven of 35 and ten of 70
   questions come back empty here instead, with a reason.
 - **Grep wins outright when you know the string.** `grep -rn "COMMON_TERM"` is
   exact, instant and complete, and nothing here replaces it.
@@ -651,6 +693,16 @@ need a dependency, and the policy for those is settled: they follow the
 embedding provider's pattern — optional, user-selected, never in the default
 install. Full reasoning in [docs/ROADMAP.md](docs/ROADMAP.md).
 
+**Whether the skill fires unprompted is not measured**, and until 1.2.1 this
+project claimed no command could measure it. That was wrong: `claude plugin
+eval` grades exactly this, with `tool_used: Skill` as a plugin-fired indicator
+and a no-plugin baseline arm. It is unmeasured because the command is in early
+access on the account here and its case schema is undocumented, so a suite
+written from `--help` fragments could not be run even once to see whether it
+loads — and a suite that silently fails to load reads as a gate while checking
+nothing. Since 1.2.0 the four commands give an entry path that does not depend
+on it.
+
 ## 12 · Development
 
 ```bash

{rag_your_code-1.2.0 → rag_your_code-1.3.0}/src/ragyourcode/__init__.py

@@ -3,4 +3,4 @@
 from .models import CodeUnit, SearchResult
 
 __all__ = ["CodeUnit", "SearchResult"]
-__version__ = "1.2.0"
+__version__ = "1.3.0"

{rag_your_code-1.2.0 → rag_your_code-1.3.0}/src/ragyourcode/config.py

@@ -201,10 +201,15 @@ SETTINGS: tuple[Setting, ...] = (
     ),
     # How much of a question has to reach the index before an answer counts as
     # evidence rather than as a guess. Measured, not chosen: across two
-    # repositories, two languages and 126 questions, 0.40 is the largest value
-    # at which every question that was being answered correctly still is, and
-    # it silences 47% of questions whose answer is not in the repository at
-    # all. It is a ratio inside the query, so unlike a score threshold it does
+    # repositories, two languages and every question set under `benchmarks/`,
+    # 0.40 is the largest value at which every question that was being answered
+    # correctly still is. On its own it silences three fifths of the questions
+    # whose answer is not in the repository at all; the rest is what
+    # `search.min_concentration` below adds. The command is the claim --
+    # `repo_queries --questions benchmarks/absent_queries.json
+    # --min-concentration 0` -- because a count typed into a comment is a figure
+    # nothing checks, and both numbers this sentence used to carry had rotted.
+    # It is a ratio inside the query, so unlike a score threshold it does
     # not move when the corpus or the scale of the ranking does -- the defect
     # that made `confidence_threshold = 0.8` stop meaning anything.
     Setting(
@@ -221,8 +226,11 @@ SETTINGS: tuple[Setting, ...] = (
     # unrelated declarations -- four of six words found in four places with
     # nothing to do with one another or with what was asked. Measured across
     # four rulers, requiring a quarter of a query's rarity to land inside one
-    # unit leaves the two rulers over undescribed code unchanged and roughly
-    # halves the questions that get an answer they should not have. Rarity-
+    # unit leaves the two rulers over undescribed code unchanged and removes
+    # most of what the coverage bar alone still answers; the ablation table
+    # carries the numbers, in docs/ROADMAP.md, rather than this line, which
+    # said "roughly halves" while that table said an order of magnitude.
+    # Rarity-
     # weighted rather than counted, because a unit holding two ordinary words is
     # not better evidence than one holding the rare word the question is about.
     Setting(

{rag_your_code-1.2.0 → rag_your_code-1.3.0}/src/ragyourcode/search.py

@@ -119,7 +119,7 @@ class Evidence:
 
     Ranking answers "which of these is best". It cannot answer "is any of this
     an answer", and reading the first as the second is what let a repository
-    reply to all 32 questions in `benchmarks/absent_queries.json` -- every one
+    reply to every question in `benchmarks/absent_queries.json` -- each one
     about a subject neither repository implements. `where are CUDA kernels
     dispatched to the device` came back with a test about word counting, on the
     evidence of `are`, `the`, `to` and `where`. That is not a Chinese problem

{rag_your_code-1.2.0 → rag_your_code-1.3.0}/tests/test_evidence.py

@@ -1,9 +1,10 @@
 """Retrieval must be able to say it has no answer.
 
 Ranking always produces a least-bad unit and returns it with a score and a
-rank, which read exactly like an answer. Graded against 32 questions about
-subjects neither this repository nor `benchmarks/cold_queries.json`'s
-repository implements, every single one came back answered -- `where are CUDA
+rank, which read exactly like an answer. Graded against every question in
+`benchmarks/absent_queries.json` -- subjects neither this repository nor
+`benchmarks/cold_queries.json`'s repository implements -- every single one came
+back answered, on both repositories. `where are CUDA
 kernels dispatched to the device` on the evidence of `are`, `the`, `to` and
 `where`. These assert the second question retrieval now asks: not which unit
 ranks highest, but whether any of this is evidence at all.

{rag_your_code-1.2.0 → rag_your_code-1.3.0}/tests/test_metadata.py

@@ -321,3 +321,35 @@ def test_every_documented_provider_block_actually_configures_that_provider(tmp_p
         # arrangement exists to prevent.
         assert "sk-" not in settings
     assert {"sentence-transformers", "openai-compatible"} <= seen, f"undocumented providers; README shows {sorted(seen)}"
+
+
+# --- a measurement nobody can re-run is a claim, not a measurement ----------
+
+BENCHMARKS = tuple(
+    sorted(
+        path.name
+        for path in (ROOT / "benchmarks").glob("*.py")
+        if path.name != "__init__.py"
+    )
+)
+
+
+def test_every_benchmark_script_is_listed_in_its_own_index():
+    """The index of the rulers is discovered, not maintained by hand.
+
+    1.3.0 found two figures in the README -- query latency, and the entire Grep
+    head-to-head -- produced by scripts that had never been committed. Nobody
+    could re-derive either, and the second one meant the phrase "a Grep loop"
+    had no definition a reader could argue with.
+
+    Both directions matter. A script absent from the index is one nobody knows
+    to run; a command in the index naming a script that does not exist is the
+    install-line defect this repository shipped twice. Discovery by glob is
+    what keeps the seventh script from being the one nothing checks.
+    """
+    index = (ROOT / "benchmarks" / "README.md").read_text(encoding="utf-8")
+    assert BENCHMARKS, "benchmarks/ holds no scripts; this guard would pass vacuously"
+    for name in BENCHMARKS:
+        assert f"benchmarks.{Path(name).stem}" in index, f"{name} is in benchmarks/ and not in its README"
+    for module in re.findall(r"benchmarks\.([a-z_]+)", index):
+        assert (ROOT / "benchmarks" / f"{module}.py").is_file(), f"the README runs benchmarks.{module}, which does not exist"