boost-skill-cli 1.2.153__tar.gz → 1.2.154__tar.gz

{boost_skill_cli-1.2.153 → boost_skill_cli-1.2.154}/CLAUDE.md

@@ -101,16 +101,23 @@ engines over the golden set (`tests/eval/golden.jsonl`) and floors **four**
 metrics on BM25 — recall@k, hit@1, MRR and nDCG@k. Flooring recall alone was a
 hole rather than a simplification: a ranker that finds the right answer every
 time and never ranks it first scores recall@10 1.000 with hit@1 0.000, and
-passed. The golden set grades real catalog items **by name**, so it needs a
-corpus: `scripts/ensure_eval_corpus.sh` first taps the pinned repo list in
-`tests/eval/taps.txt` (the minimal set covering all golden targets — `boost tap
---defaults` is NOT enough, it omits every rule/workflow repo). The list is
-**twenty** repos: the first six cover every golden target, the rest exist so the
-corpus is a realistic size. That matters more than it sounds — over the six
-alone (921 entries at the current pins) BM25 scores 0.989 / 0.769 / 0.848 /
-0.880, and over the twenty (10,731) it scores **0.841 / 0.484 / 0.607 / 0.655**
-on the same golden set (what `tests/eval/baseline.json` records), so three of
-the four old floors fail once the corpus stops being tiny.
+passed. The golden set grades real catalog items **by the body they name**, so it
+needs a corpus: `scripts/ensure_eval_corpus.sh` first taps the pinned repo list in
+`tests/eval/taps.txt` (`boost tap --defaults` is NOT enough, it omits every
+rule/workflow repo). The list is **twenty** repos and **not one of them
+can be dropped**: the first six are the set the name-graded floors were first
+measured on, and the `# --- scale` divider below them no longer separates
+"targets" from "filler" — `golden.jsonl` pins exemplars into two repos below
+it and `golden-natural.jsonl` into five, so no proper subset of the file can
+score either set. The rest exist so the corpus is a realistic size. That matters more than it sounds — over the six
+alone (921 entries at the current pins) BM25 scored 0.989 / 0.769 / 0.848 /
+0.880, and over the twenty (10,731) it scores **0.846 / 0.484 / 0.607 / 0.659**
+(what `tests/eval/baseline.json` records), so three of the four old floors fail
+once the corpus stops being tiny. The six-repo row is **historical**: it was
+measured name-graded, before the set was pinned, and cannot be re-run against
+today's `golden.jsonl` — see the coupling rule below. It stays re-derivable:
+`git show e11a34f2:tests/eval/golden.jsonl` scored over a six-tap corpus
+reproduces it exactly.
 
 **The ranked list de-duplicates on the content hash, not the name.** A grade key
 decides both relevance and identity, and keying identity on the name collapsed
@@ -118,16 +125,22 @@ decides both relevance and identity, and keying identity on the name collapsed
 ranker with a compression that existed only in the scoring code, and worth about
 one query of recall@10. That is where the old "recall is 1.000" folklore came
 from; the six-repo corpus measured 0.978 at the pins of the time once mirrors
-collapse and homonyms do not. Relevance is still decided by name (or by content
-class when a golden row pins an `exemplar`), so the sets can migrate a row at a
-time.
+collapse and homonyms do not. Relevance is decided by the content class of the
+bodies a row pins in its `exemplar`, or by name for a row that pins none. The
+two conventions coexist so a set can migrate a row at a time, and **both
+shipped sets have now finished** — `golden.jsonl` 91/91 here, and
+`golden-natural.jsonl` 50/50 back in #434. So no shipped set exercises the
+name path any more: it is kept for a set added outside this repo, and unit
+tests (`TestNameGradingIsUnchanged`) are the only thing covering it. Don't
+delete it on the strength of the coverage report, and don't claim a shipped
+set still uses it.
 
 The floors are fixed absolute values, and the uniform "~10%" margin this file
 claimed from 170d52c0 (2026-07-31) onward was never one of them. They sit under
-their measured values (7.2% / 17.3% / 14.3% / 11.5%), recall@k the tightest —
+their measured values (7.8% / 17.3% / 14.3% / 12.1%), recall@k the tightest —
 loose enough that upstream drift can't flake the build, tight enough to catch a
-collapse, but a 2.39x spread rather than one number, and the tightest of them
-is worth 5.5 queries of 91. The spread is the design working, not drifting: the
+collapse, but a 2.21x spread rather than one number, and the tightest of them
+is worth 6.0 queries of 91. The spread is the design working, not drifting: the
 monthly refresh re-measures the row and deliberately leaves the floors where
 they are, so every refresh that improves a metric widens that metric's margin.
 Floors are re-stated after a refresh, never re-derived — a floor that follows
@@ -156,6 +169,25 @@ floored recall alone at 0.85 against a measured 0.863 — a buffer of 1.15
 queries out of 91 — and applied none of the other three). Moving a pin means
 regenerating the baseline; the file says how.
 
+**A pinned golden row couples the set to a corpus, and that is the trade.**
+An `exemplar` is `tap::skill_md`, so a row can only be scored over a corpus
+holding that tap: `prepare_row` raises `SystemExit` on a spec it cannot
+resolve, deliberately, because a silent fall back to name grading turns a typo
+into a quietly weaker gate that still reports a number. Consequences:
+
+- **A subset corpus can no longer score `golden.jsonl`.** The six-repo
+  comparison figure is historical for this reason, not stale. `taps-scale.txt`
+  is fine — it holds every row of `taps.txt` and 129 more, so
+  `eval-scale.yml` keeps scoring the same set.
+- **A pin move can invalidate a path.** `eval-corpus-refresh.yml`'s re-baseline
+  step is not `continue-on-error`, so a renamed file stops the monthly refresh
+  before its pull request opens. That is the intended signal and the fix is to
+  re-resolve the path in the new clone — `--worksheet` will not help, because
+  it lists rows still graded **by name** and a pinned row is not one.
+- **Do not "fix" it by tolerating an unresolvable spec in a multi-spec list.**
+  A refresh that drops a repo would then narrow every such row's class to
+  whatever still resolves, which is the same silent weakening one level down.
+
 **Baselines are keyed by query set** (`name@content-digest`), so one file holds
 both `golden.jsonl` and `golden-natural.jsonl` without either overwriting the
 other. Before that, running the natural-language set printed eight confident

{boost_skill_cli-1.2.153 → boost_skill_cli-1.2.154}/Makefile

@@ -122,16 +122,21 @@ dist-check:
 # 0.000 passed.
 #
 # The floors are calibrated against a TWENTY-tap corpus, not the six it used to
-# be. Over the six (921 entries) BM25 scores 0.989 / 0.769 / 0.848 / 0.880;
-# over twenty it scores 0.841 / 0.484 / 0.607 / 0.655 on the same golden set,
-# which is what tests/eval/baseline.json records at the current pins. The
+# be. Over the six (921 entries) BM25 scored 0.989 / 0.769 / 0.848 / 0.880;
+# over twenty it scores 0.846 / 0.484 / 0.607 / 0.659, which is what
+# tests/eval/baseline.json records at the current pins. Not the same golden
+# set: the six-repo row is name-graded and HISTORICAL, measured before every
+# row pinned an exemplar, and today's golden.jsonl cannot be scored over six
+# taps at all (a pin naming a tap outside the corpus is a hard error). It is
+# still re-derivable -- `git show e11a34f2:tests/eval/golden.jsonl` over a
+# six-tap corpus reproduces it to three places. The
 # second set is what a real user sees, so flooring against the first was
 # measuring the corpus rather than the retrieval — three of the four old floors
 # fail outright once the corpus is realistic. The four floors sit a margin
-# under their measured values (7.2% / 17.3% / 14.3% / 11.5%), recall@k the
-# tightest at 5.5 queries of 91: loose enough that a query or two cannot flake
+# under their measured values (7.8% / 17.3% / 14.3% / 12.1%), recall@k the
+# tightest at 6.0 queries of 91: loose enough that a query or two cannot flake
 # the build, tight enough that a collapse fails it. That margin is not uniform
-# and never was -- it is a 2.39x spread, because the row moves with every
+# and never was -- it is a 2.21x spread, because the row moves with every
 # monthly refresh and the floors deliberately do not. So it is quoted rather
 # than asserted, and tests/unit/test_corpus_prose.py fails until the new
 # spread is written here. Regression-vs-baseline stays relaxed

{boost_skill_cli-1.2.153 → boost_skill_cli-1.2.154}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.2.153
+Version: 1.2.154
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License-Expression: Apache-2.0

{boost_skill_cli-1.2.153 → boost_skill_cli-1.2.154}/boost_cli/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '1.2.153'
-__version_tuple__ = version_tuple = (1, 2, 153)
+__version__ = version = '1.2.154'
+__version_tuple__ = version_tuple = (1, 2, 154)
 
 __commit_id__ = commit_id = None

{boost_skill_cli-1.2.153 → boost_skill_cli-1.2.154}/boost_skill_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.2.153
+Version: 1.2.154
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License-Expression: Apache-2.0

{boost_skill_cli-1.2.153 → boost_skill_cli-1.2.154}/docs/eval.html

@@ -146,7 +146,7 @@
 
     <div class="stats">
       <div class="stat reveal"><b>5</b><span>Evaluation tiers</span></div>
-      <div class="stat reveal"><b>0.841</b><span>BM25 recall@10 · pinned corpus</span></div>
+      <div class="stat reveal"><b>0.846</b><span>BM25 recall@10 · pinned corpus</span></div>
       <div class="stat reveal"><b>141</b><span>Golden judgments</span></div>
       <div class="stat reveal"><b>0</b><span>Deps in the required gate</span></div>
     </div>
@@ -326,7 +326,8 @@
         regression &mdash; it is what users already had, now visible.</p>
 
       <h3>2 &middot; The score was also about the shape of the question</h3>
-      <p>The keyword golden set grades catalog items <b>by name</b>, so its queries carry the
+      <p>The keyword golden set was graded <b>by name</b> when these runs were made &mdash; both
+        shipped sets now pin an exemplar body on every row &mdash; and its queries carry the
         vocabulary of the thing they are looking for. That is BM25's home ground. The 50 queries in
         <code>golden-natural.jsonl</code> describe a <em>problem</em> instead &mdash; each written
         from its target's own description, with the target's distinctive name tokens deliberately
@@ -341,7 +342,8 @@
         query sets, <code>k=10</code>. <b>Voyage is absent: it needs an API key, and these runs were
         keyless</b>, which is the configuration most users are in.</p>
 
-      <p><b>Keyword golden set</b> &mdash; 91 queries that grade items by name:</p>
+      <p><b>Keyword golden set</b> &mdash; 91 queries, graded by name at the pins these runs
+        used; every row pins an exemplar body today:</p>
       <table class="contract">
         <thead><tr><th>engine</th><th>recall@10</th><th>hit@1</th><th>MRR</th><th>nDCG@10</th></tr></thead>
         <tbody>
@@ -397,8 +399,10 @@
     <div class="wrap">
       <span class="eyebrow">Sequence · the required gate</span>
       <h2 class="sec-title">How Tier 1 runs on every PR</h2>
-      <p class="lead" style="margin-top:20px">The golden set grades items <em>by name</em>, so the
-        repos that hold them must be present. CI taps a pinned corpus first, builds the BM25 index,
+      <p class="lead" style="margin-top:20px">Every golden row pins the <em>body</em> it was
+        written about, as <code>tap::skill_md</code>, so the repos that hold them must be present
+        &mdash; a pin the corpus cannot resolve halts the gate and names the missing spec,
+        rather than quietly scoring a smaller set. CI taps a pinned corpus first, builds the BM25 index,
         scores the golden set, and floors recall — all offline after the tap, all deterministic.</p>
 
       <div class="diagram">

{boost_skill_cli-1.2.153 → boost_skill_cli-1.2.154}/docs/roadmap/items/gated-golden-set-has-zero-exemplars.md

@@ -1,17 +1,17 @@
 ---
 id: gated-golden-set-has-zero-exemplars
 board: code
-section: planned
-status: planned
+section: shipped
+status: shipped
 category: Quality · Retrieval eval
 complexity: M
 impact: Med
 wow: 3
 note: CLAUDE.md presents exemplar pinning as a live migration — "relevance is still decided…
 order: 214
-owner:
-pr:
-title: The exemplar mechanism was applied to the ungated set only: golden.jsonl is 0/91 pinned, and 10 of its 43 hit@1 credits are on names the metric cannot adjudicate
+owner: loop/gated-golden-exemplars
+pr: "994"
+title: The exemplar mechanism was applied to the ungated set only: golden.jsonl is 0/91 pinned, and 10 of its 44 hit@1 credits are on names the metric cannot adjudicate
 ---
 <b>Measured.</b> On the 10,731-entry corpus that committed <code>taps.txt</code> materializes today, <code>tests/eval/golden.jsonl</code> — the only set the required <code>eval</code> gate floors — is 0/91 exemplar-pinned while the ungated <code>golden-natural.jsonl</code> is 50/50; 10 of its 44 hit@1 credits (22.7%) are awarded on a name that resolves to more than one distinct body, and the hit@1 floor's entire headroom is 7.60 queries, smaller than the 10 credits the metric cannot adjudicate.
 
@@ -19,7 +19,7 @@ title: The exemplar mechanism was applied to the ungated set only: golden.jsonl
 
 <code>cd &lt;repo&gt;</code><br>
 <code>export BOOST_HOME=$TMPDIR/eval-home</code><br>
-<code># 0. confirm which corpus you are on (expect 10152)</code><br>
+<code># 0. confirm which corpus you are on (expect 10731)</code><br>
 <code>.venv/bin/python -c "import sys;sys.path.insert(0,'.');from boost_cli.core import catalog;print(len(catalog.all_entries()))"; echo EXIT=$?</code><br>
 <code># 1. exemplar counts per set</code><br>
 <code>python3 -c "</code><br>
@@ -52,6 +52,16 @@ Nothing else in the finding is wrong. 0/91 vs 50/50, the 27 undecided rows / 62
 
 5. NOT A DEFECT IN THE HARNESS: <code>exemplar_worksheet</code> (scripts/eval_retrieval.py:219) and exemplar grading work correctly and fail loudly on a bad pin.
 
-<b>Why it is worth doing.</b> golden.jsonl is the set the required <code>eval</code> gate floors, so it is the only one that can block a merge. 23.3% of its hit@1 credits are awarded on a name that maps to several genuinely different skills, and the floor's whole margin (6.60 queries) is narrower than the un-adjudicated credit count (10). That does not mean retrieval is worse than reported — the realistic shift is 1-2 queries — it means the published margin cannot be read as precision about the intended skill, so anyone tuning blend weights or pool depth against this gate inherits an unquantified slack.
+<b>Why it is worth doing.</b> golden.jsonl is the set the required <code>eval</code> gate floors, so it is the only one that can block a merge. 22.7% of its hit@1 credits are awarded on a name that maps to several genuinely different skills, and the floor's whole margin (7.60 queries) is narrower than the un-adjudicated credit count (10). That does not mean retrieval is worse than reported — the realistic shift is 1-2 queries — it means the published margin cannot be read as precision about the intended skill, so anyone tuning blend weights or pool depth against this gate inherits an unquantified slack.
 
 <em>Found by an automated audit of retrieval/eval quality, the search &amp; browse surfaces, and first-run onboarding; every finding was then re-measured from scratch by an independent adversarial verifier whose instruction was to refute it. Verdict: <b>CORRECTED</b>. No fix is prescribed here — the measurement is the contribution.</em>
+
+<b>Shipped.</b> All 91 rows of <code>tests/eval/golden.jsonl</code> now pin an <code>exemplar</code>, so the required gate grades by content class rather than by name. <code>--worksheet</code> reports 0 rows still name-graded, and <code>prepare_row</code> raises on a spec that resolves to nothing, so a stale pin fails the gate rather than quietly weakening it.
+
+<b>The pins are a judgment, so they were made as one.</b> 64 of the 91 rows resolved to a single body and were pinned by lookup. Clustering the remaining 27 rows' candidate bodies mechanically (difflib ratio 0.90) collapsed 7 of them to a single re-publication cluster, pinnable as a list by <code>diff</code> rather than opinion. That left <b>20</b> genuine homonym rows, each judged by an agent under one rule — keep a cluster if a user typing this query would be correctly served by it; two independent implementations of the same job both count; drop only a cluster doing a different job — and each judgment then attacked by two adversarial verifiers, one hunting over-narrow pins and one hunting over-broad. 60 agents, <b>0 of 20 decisions refuted</b>, 19 of 20 at high confidence. Two clusters were dropped: Sentry's <code>brand-guidelines</code> (copy-writing, not a visual brand system) and oh-my-agent's <code>pdf</code> (PDF&rarr;Markdown conversion, which cannot fill a form field).
+
+<b>What the measurement did, and did not, do.</b> Scored per query against the old set and the new on the same 10,731-entry corpus, <b>exactly 1 of 91 rows changed</b>. The other 90 grade identically under name and content class, which is the honest result: on this corpus the name was already selecting the right body almost everywhere, and the value of pinning is that it can no longer stop doing so silently. The one mover is the set's only two-name row (<code>style an html artifact with a color and font theme</code>, relevant <code>theme-factory</code> + <code>web-artifacts-builder</code>): <code>relevant</code> is documented as names that <em>count as a correct top hit</em> — alternatives — but <code>recall@k</code> over a two-element set scored 0.5 for finding one of them. The four pinned bodies are one class, so it scores 1.0, and the aggregate moved 76.5/91 &rarr; 77.0/91. <code>hit@1</code> and <code>MRR</code> are bit-identical.
+
+BM25 over the twenty taps: 0.841 / 0.484 / 0.607 / 0.655 &rarr; <b>0.846 / 0.484 / 0.607 / 0.659</b>. All four floors still clear, and none was moved &mdash; margins are now 7.8% / 17.3% / 14.3% / 12.1%, a 2.21x spread, recall@k binding at 6.0 queries of 91.
+
+<b>The cost, recorded rather than discovered later.</b> A pin is <code>tap::skill_md</code>, so the set can only be scored over a corpus holding that tap. The six-repo comparison figure in <code>Makefile</code> is therefore <b>historical</b>, not stale: today's <code>golden.jsonl</code> cannot be scored over six taps at all. It stays re-derivable &mdash; <code>git show e11a34f2:tests/eval/golden.jsonl</code> over a six-tap corpus reproduces 0.989 / 0.769 / 0.848 / 0.880 to three places. <code>taps-scale.txt</code> is unaffected: it is a superset of <code>taps.txt</code> (149 rows &supe; 20), so <code>eval-scale.yml</code> keeps scoring the same set. The real exposure is <code>eval-corpus-refresh.yml</code>, whose re-baseline step is not <code>continue-on-error</code>: a monthly pin move past a renamed file now stops 141 pinned rows instead of 50, and the pull request does not open. That is the intended signal and is written up as a non-obvious rule in <code>CLAUDE.md</code>.

{boost_skill_cli-1.2.153 → boost_skill_cli-1.2.154}/docs/roadmap/items/required-eval-gate-still-grades-by-name.md

@@ -1,16 +1,16 @@
 ---
 id: required-eval-gate-still-grades-by-name
 board: code
-section: planned
-status: planned
+section: shipped
+status: shipped
 category: Tech-debt
 complexity: M
 impact: Med
 wow: 3
 note: CLAUDE.md states "Relevance is still decided by name (or by content class when a gold…
 order: 243
-owner:
-pr:
+owner: loop/gated-golden-exemplars
+pr: "994"
 title: The exemplar migration never reached the query set the required gate runs: 0 of 91 rows, and 27 of them have an ambiguous target on the gate's own corpus
 ---
 <b>Measured.</b> Both invocations of the merge-blocking retrieval gate — the Makefile <code>eval</code> target and ci.yml's "retrieval quality gate" step — run <code>scripts/eval_retrieval.py</code> with no <code>--golden</code>, so both take <code>DEFAULT_GOLDEN = tests/eval/golden.jsonl</code>; 0 of that file's 91 rows carry an <code>exemplar</code>, and on the gate's own 20-tap corpus 27 of the 91 rows have at least one target name that resolves to more than one content digest (21 of 77 distinct target names are ambiguous).
@@ -40,3 +40,11 @@ Floor-margin exposure, with that qualification attached: hit@1 measures 44/91 =
 <b>Why it is worth doing.</b> This is the only gate in <code>make check</code> that can fail a merge on retrieval quality, and every retrieval decision validated against it (RRF fusion vs preferring dense, pool depth, whether the LLM rerank earns its keep) inherits the looseness. The project already established the fix and built the harness for it; the required set simply never got migrated, and because the card is closed as shipped nothing tracks the remaining work on the set that matters most.
 
 <em>Found by an automated audit of retrieval/eval quality, the search &amp; browse surfaces, and first-run onboarding; every finding was then re-measured from scratch by an independent adversarial verifier whose instruction was to refute it. Verdict: <b>CONFIRMED</b>. No fix is prescribed here — the measurement is the contribution.</em>
+
+<b>Shipped, with the card above.</b> This card and <code>gated-golden-set-has-zero-exemplars</code> measured one defect from two angles — 0/91 pinned rows, and 27 of those rows ambiguous on the gate's own corpus — and one change closes both. All 91 rows of <code>tests/eval/golden.jsonl</code> now pin an <code>exemplar</code>; the required gate grades by content class.
+
+<b>This card's severity call was right, and the measurement confirms it.</b> The verifier lowered severity from High to Medium on the grounds that most ambiguous names are re-publications of one upstream skill, so a "wrong body" was still a correct answer. That is exactly what the pins found: the 20 judged rows kept 2-4 bodies each and dropped only <b>2</b> clusters in total, and scoring per query showed <b>90 of 91 rows grade identically</b> under name and content class. The looseness was real and worth closing, and it was not inflating the published numbers.
+
+The card's own row arithmetic predicted the result before the pins existed: it stated <em>"recall@10 measures 77/91 = 0.846 against a floor needing 71 rows (margin 6 rows)"</em>, counting a row as matched or not. The metric averaged 76.5/91 = 0.841 because the set's one two-name row scored 0.5 for finding one of two documented alternatives. Pinning makes those alternatives one class, so the metric now reads 77/91 = <b>0.846</b> — the card's figure, arrived at independently.
+
+<b>What was NOT done, deliberately.</b> No floor was moved. The four floors stay at 0.78 / 0.40 / 0.52 / 0.58 and all four still clear; #993 established in this same file that a floor is restated after a measurement, never re-derived from it. The name-grading path is kept, not deleted &mdash; but nothing shipped exercises it any more: <code>golden-natural.jsonl</code> finished migrating in #434 and is 50/50, so with golden.jsonl at 91/91 both shipped sets are fully pinned. It stays for a query set added outside this repo, covered by <code>TestNameGradingIsUnchanged</code> alone.

{boost_skill_cli-1.2.153 → boost_skill_cli-1.2.154}/docs/roadmap.html

@@ -140,9 +140,9 @@
   <section class="block reveal">
     <div class="stats">
       <!-- ROADMAP:stats -->
-      <div class="stat"><b>25</b><span>Shipped</span></div>
+      <div class="stat"><b>27</b><span>Shipped</span></div>
       <div class="stat"><b>1</b><span>Next up</span></div>
-      <div class="stat"><b>76</b><span>Planned</span></div>
+      <div class="stat"><b>74</b><span>Planned</span></div>
       <div class="stat"><b>505</b><span>Loop finds</span></div>
       <!-- /ROADMAP:stats -->
     </div>
@@ -432,6 +432,30 @@
         </div>
       </article>
 
+      <article class="cap rcard" id="gated-golden-set-has-zero-exemplars">
+        <div class="head"><span class="pill shipped">Shipped</span><span class="cat">Quality · Retrieval eval</span></div>
+        <h3>The exemplar mechanism was applied to the ungated set only: golden.jsonl is 0/91 pinned, and 10 of its 44 hit@1 credits are on names the metric cannot adjudicate</h3>
+        <p class="cardbody"><a class="writeup" href="https://github.com/jonnyeclectic/boost/blob/main/docs/roadmap/items/gated-golden-set-has-zero-exemplars.md">Write-up &middot; gated-golden-set-has-zero-exemplars.md</a></p>
+        <div class="meta">
+          <span class="m">Complexity <b>M</b></span>
+          <span class="m">Impact <b>Med</b></span>
+          <span class="m">Wow <b class="wow">★★★</b></span>
+          <span class="m">CLAUDE.md presents exemplar pinning as a live migration — "relevance is still decided…</span>
+        </div>
+      </article>
+
+      <article class="cap rcard" id="required-eval-gate-still-grades-by-name">
+        <div class="head"><span class="pill shipped">Shipped</span><span class="cat">Tech-debt</span></div>
+        <h3>The exemplar migration never reached the query set the required gate runs: 0 of 91 rows, and 27 of them have an ambiguous target on the gate's own corpus</h3>
+        <p class="cardbody"><a class="writeup" href="https://github.com/jonnyeclectic/boost/blob/main/docs/roadmap/items/required-eval-gate-still-grades-by-name.md">Write-up &middot; required-eval-gate-still-grades-by-name.md</a></p>
+        <div class="meta">
+          <span class="m">Complexity <b>M</b></span>
+          <span class="m">Impact <b>Med</b></span>
+          <span class="m">Wow <b class="wow">★★★</b></span>
+          <span class="m">CLAUDE.md states "Relevance is still decided by name (or by content class when a gold…</span>
+        </div>
+      </article>
+
       <article class="cap rcard" id="install-cannot-tell-a-skill-needs-another-one">
         <div class="head"><span class="pill shipped">Shipped</span><span class="cat">Install · Research</span></div>
         <h3>Research: how <code>boost install</code> could tell that a skill needs something else installed first</h3>
@@ -1256,59 +1280,6 @@ NOT ALREADY CARDED, but read the existing card first.
         </div>
       </article>
 
-      <article class="cap rcard" id="gated-golden-set-has-zero-exemplars">
-        <div class="head"><span class="pill planned">Planned</span><span class="cat">Quality · Retrieval eval</span></div>
-        <h3>The exemplar mechanism was applied to the ungated set only: golden.jsonl is 0/91 pinned, and 10 of its 43 hit@1 credits are on names the metric cannot adjudicate</h3>
-        <p><b>Measured.</b> On the 10,731-entry corpus that committed <code>taps.txt</code> materializes today, <code>tests/eval/golden.jsonl</code> — the only set the required <code>eval</code> gate floors — is 0/91 exemplar-pinned while the ungated <code>golden-natural.jsonl</code> is 50/50; 10 of its 44 hit@1 credits (22.7%) are awarded on a name that resolves to more than one distinct body, and the hit@1 floor's entire headroom is 7.60 queries, smaller than the 10 credits the metric cannot adjudicate.
-
-<b>Reproduce it.</b>
-
-<code>cd &lt;repo&gt;</code><br>
-<code>export BOOST_HOME=$TMPDIR/eval-home</code><br>
-<code># 0. confirm which corpus you are on (expect 10152)</code><br>
-<code>.venv/bin/python -c "import sys;sys.path.insert(0,'.');from boost_cli.core import catalog;print(len(catalog.all_entries()))"; echo EXIT=$?</code><br>
-<code># 1. exemplar counts per set</code><br>
-<code>python3 -c "</code><br>
-<code>import json</code><br>
-<code>for f in ['tests/eval/golden.jsonl','tests/eval/golden-natural.jsonl']:</code><br>
-<code>    r=[json.loads(l) for l in open(f) if l.strip() and not l.startswith('#')]</code><br>
-<code>    print(f,'rows:',len(r),'with exemplar:',sum(1 for x in r if x.get('exemplar')))"; echo EXIT=$?</code><br>
-<code># 2. the harness's own list of undecided gated rows</code><br>
-…
-
-<b>What adversarial verification corrected.</b> This card's numbers are the re-measured ones, not the ones first reported.
-
-The defect is real and the spine is exact, but five figures are stale by one corpus refresh. The finding measured on the 10,152-entry corpus CLAUDE.md documents; committed <code>tests/eval/taps.txt</code> moved on 2026-09-01 (cbc0a58b) and now materializes 10,731 entries, which is what <code>make eval</code>/CI builds today. Corrected values, all measured by me on that corpus:
-
-- hit@1 credits: 43/91 = 0.473 -&gt; <b>44/91 = 0.484</b> - un-adjudicated share of credits: 23.3% -&gt; <b>22.7%</b> (the count stays <b>10</b>) - floor headroom: 6.60 queries -&gt; <b>7.60 queries</b> - worst-case bound stripping all 10: 33/91 = 0.363 -&gt; <b>34/91 = 0.374</b> (still below the 0.400 floor) - BM25 four-metric line: 0.852 / 0.473 / 0.605 / 0.657 -&gt; <b>0.841 / 0.484 / 0.607 / 0.655</b>
-
-The finding's numbers are correct for the corpus it names and it disclosed the provenance honestly; they are nonetheless the wrong numbers to print about "the set the required eval gate floors", because the gate no longer builds that corpus. A card must publish the 10,731 figures and name the corpus.
-
-Nothing else in the finding is wrong. 0/91 vs 50/50, the 27 undecided rows / 62 candidate bodies, the same 10 names and 10 queries, the skill-creator worked example, the not-carded check, and the "realistic shift is 1-2 queries, the floor still passes" bound all hold on BOTH corpora.
-
-<b>What is NOT established.</b> Recorded because a card that overstates its own evidence is worse than no card.
-
-1. WHAT IS CORPUS-INDEPENDENT (identical on 10,152 and 10,731): exemplar counts 0/91 and 50/50; 27 undecided worksheet rows over 62 candidate bodies; the same 10 ambiguous-credit names and the same 10 queries; the skill-creator example. Only the credit count and the two ratios moved. Build the card on the invariant half and quote the 10,731 figures for the rest.
-
-2. TRAP FOR THE NEXT VERIFIER: the shared read-only <code>$TMPDIR/eval-home</code> is a PRE-REFRESH corpus — its <code>.eval-corpus-ready</code> sentinel digest does not match <code>sha256(tests/eval/taps.txt)</code>, and 10 of its 20 clones sit at a different commit. Anyone who measures only there reproduces the finding's old numbers verbatim and marks it CONFIRMED without noticing. Rebuilding at the committed pins in a disposable HOME took ~3 minutes and is what produced the correction.
-
-3. DO NOT COPY Makefile:126, which still comments "over twenty it scores 0.863 / 0.473 / 0.607 / 0.662". That matches neither corpus (10,152 gives 0.852/0.473/0.605/0.657; 10,731 gives 0.841/0.484/0.607/0.655). CLAUDE.md:67's figures are right for the old corpus and now also stale.
-
-4. SCOPE OF MY REPRO: I verified the BM25 engine only (<code>--engines bm25</code>), which is what the gate floors; I did not build a dense store (impossible here). I did not run <code>make eval</code> end to end, only its second command against a corpus I materialised with its first.
-
-5. NOT A DEFECT IN THE HARNESS: <code>exemplar_worksheet</code> (scripts/eval_retrieval.py:219) and exemplar grading work correctly and fail loudly on a bad pin.
-
-<b>Why it is worth doing.</b> golden.jsonl is the set the required <code>eval</code> gate floors, so it is the only one that can block a merge. 23.3% of its hit@1 credits are awarded on a name that maps to several genuinely different skills, and the floor's whole margin (6.60 queries) is narrower than the un-adjudicated credit count (10). That does not mean retrieval is worse than reported — the realistic shift is 1-2 queries — it means the published margin cannot be read as precision about the intended skill, so anyone tuning blend weights or pool depth against this gate inherits an unquantified slack.
-
-<em>Found by an automated audit of retrieval/eval quality, the search &amp; browse surfaces, and first-run onboarding; every finding was then re-measured from scratch by an independent adversarial verifier whose instruction was to refute it. Verdict: <b>CORRECTED</b>. No fix is prescribed here — the measurement is the contribution.</em></p>
-        <div class="meta">
-          <span class="m">Complexity <b>M</b></span>
-          <span class="m">Impact <b>Med</b></span>
-          <span class="m">Wow <b class="wow">★★★</b></span>
-          <span class="m">CLAUDE.md presents exemplar pinning as a live migration — "relevance is still decided…</span>
-        </div>
-      </article>
-
       <article class="cap rcard" id="info-deps-materialize-a-dir-they-then-reject">
         <div class="head"><span class="pill shipped">Shipped</span><span class="cat">UX · Bug</span></div>
         <h3><code>boost info</code>/<code>deps</code> on a not-installed rule or workflow widens the tap's sparse cone for a directory <code>source_dir_for</code> immediately rejects</h3>
@@ -1633,44 +1604,6 @@ Nothing else in the finding is wrong. 0/91 vs 50/50, the 27 undecided rows / 62
         </div>
       </article>
 
-      <article class="cap rcard" id="required-eval-gate-still-grades-by-name">
-        <div class="head"><span class="pill planned">Planned</span><span class="cat">Tech-debt</span></div>
-        <h3>The exemplar migration never reached the query set the required gate runs: 0 of 91 rows, and 27 of them have an ambiguous target on the gate's own corpus</h3>
-        <p><b>Measured.</b> Both invocations of the merge-blocking retrieval gate — the Makefile <code>eval</code> target and ci.yml's "retrieval quality gate" step — run <code>scripts/eval_retrieval.py</code> with no <code>--golden</code>, so both take <code>DEFAULT_GOLDEN = tests/eval/golden.jsonl</code>; 0 of that file's 91 rows carry an <code>exemplar</code>, and on the gate's own 20-tap corpus 27 of the 91 rows have at least one target name that resolves to more than one content digest (21 of 77 distinct target names are ambiguous).
-
-<b>Reproduce it.</b>
-
-<code>cd &lt;repo&gt;</code><br>
-<code>python3 -c "import json; rows=[json.loads(l) for l in open('tests/eval/golden.jsonl') if l.strip() and not l.startswith('#')]; print(len(rows),'rows;',sum(1 for r in rows if r.get('exemplar')),'with exemplar')"</code><br>
-<code>python3 -c "import json; rows=[json.loads(l) for l in open('tests/eval/golden-natural.jsonl') if l.strip() and not l.startswith('#')]; print(len(rows),'rows;',sum(1 for r in rows if r.get('exemplar')),'with exemplar')"</code><br>
-<code>grep -n 'DEFAULT_GOLDEN' scripts/eval_retrieval.py</code><br>
-<code>sed -n '134,139p' Makefile</code><br>
-<code>sed -n '269,279p' .github/workflows/ci.yml</code><br>
-<code>export BOOST_HOME=$TMPDIR/eval-home</code><br>
-<code>python3 -c "</code><br>
-<code>import json,glob,collections,os</code><br>
-<code>n2c=collections.defaultdict(set)</code><br>
-…
-
-<b>Verification found nothing to correct.</b> Every number, <code>file:line</code> and command output above was independently re-derived and matched exactly.
-
-<b>What is NOT established.</b> Recorded because a card that overstates its own evidence is worse than no card.
-
-Severity lowered from the claim's High to Medium on a measurement the claim did not make. I inspected the bodies behind the ambiguous names and they are two different things. Genuine homonyms exist — <code>brand-guidelines</code> is Anthropic brand colors vs OpenAI brand colors vs Sentry *copy writing* (three different jobs), and <code>pdf</code> is a general PDF skill vs a workflow that converts PDF to Markdown with one specific tool — but they are a minority, and neither appears among the rank-1 hits. The names that actually score hit@1 through an ambiguous target (skill-creator, slack-gif-creator, theme-factory, prompt-engineering, mcp-builder) are re-publications of one upstream skill across anthropics/skills, composio-community/awesome-codex-skills and sickn33/antigravity-awesome-skills, differing by a few bytes. A "wrong body" there is still a correct answer to the user, so the looseness is much smaller in practice than in principle. I did not adopt the shipped card's "the looseness was latent" finding as a reason: that was measured at hit@1 0.160, where there was nothing for ambiguity to inflate, whereas this set measures 0.484 — a different regime, so it does not transfer.
-
-Floor-margin exposure, with that qualification attached: hit@1 measures 44/91 = 0.484 against a floor needing 37 rows (margin 7 rows) with 10 rank-1 hits landing on an ambiguous name; recall@10 measures 77/91 = 0.846 against a floor needing 71 rows (margin 6 rows) with 20 rows matched only via ambiguous names.
-
-<b>Why it is worth doing.</b> This is the only gate in <code>make check</code> that can fail a merge on retrieval quality, and every retrieval decision validated against it (RRF fusion vs preferring dense, pool depth, whether the LLM rerank earns its keep) inherits the looseness. The project already established the fix and built the harness for it; the required set simply never got migrated, and because the card is closed as shipped nothing tracks the remaining work on the set that matters most.
-
-<em>Found by an automated audit of retrieval/eval quality, the search &amp; browse surfaces, and first-run onboarding; every finding was then re-measured from scratch by an independent adversarial verifier whose instruction was to refute it. Verdict: <b>CONFIRMED</b>. No fix is prescribed here — the measurement is the contribution.</em></p>
-        <div class="meta">
-          <span class="m">Complexity <b>M</b></span>
-          <span class="m">Impact <b>Med</b></span>
-          <span class="m">Wow <b class="wow">★★★</b></span>
-          <span class="m">CLAUDE.md states "Relevance is still decided by name (or by content class when a gold…</span>
-        </div>
-      </article>
-
       <article class="cap rcard" id="stale-tap-hint-dead-for-tap-only-installs">
         <div class="head"><span class="pill shipped">Shipped</span><span class="cat">Tech-debt</span></div>
         <h3>The "taps last refreshed N days ago" hint can never fire on a machine that tapped and never ran <code>boost update</code> — the only writer of the marker is <code>update</code> itself</h3>

{boost_skill_cli-1.2.153 → boost_skill_cli-1.2.154}/scripts/eval_corpus.py

@@ -181,7 +181,10 @@ MAX_CONTENT_SHARE = 0.40
 # taps-scale.txt is generated, and its generator owns its header.
 SIZE_START = "# --- corpus size: written from the rows below by eval_corpus.py ---"
 SIZE_END = "# --- end corpus size ---"
-# The rows above this line hold every golden target; the rest are scale.
+# The rows above this line are the six the name-graded floors were first
+# measured on. They are NOT "every golden target" any more: both shipped
+# query sets pin exemplars below the divider too, so the label written from
+# it says "six-repo set", not "targets". taps.txt says why at length.
 SCALE_DIVIDER = "# --- scale"
 
 
@@ -348,8 +351,10 @@ def size_lines(text: str) -> list[str]:
     head, divider, _tail = text.partition("\n" + SCALE_DIVIDER)
     if divider:
         targets = [n for _r, _s, n in parse_taps(head) if n is not None]
-        lines.append("# targets: %s entries in the %d repos above the scale "
-                     "divider" % (f"{sum(targets):,}", len(targets)))
+        lines.append("# sixrepo: %s entries in the %d repos above the scale"
+                     % (f"{sum(targets):,}", len(targets)))
+        lines.append("#          divider -- a historical baseline, not a "
+                     "subset that scores")
     # strict=False: a list of one repo has a largest and no runner-up.
     labels = ("largest:", "next:   ")
     for label, (repo, count, share) in zip(labels, ranked, strict=False):

{boost_skill_cli-1.2.153 → boost_skill_cli-1.2.154}/scripts/eval_retrieval.py

@@ -140,8 +140,13 @@ METRICS: dict[str, Callable[[Sequence[str], set, int], float]] = {
 # punish a correct answer for arriving from a mirror), while a different skill
 # sharing the name does not.
 #
-# Rows without an exemplar still decide RELEVANCE by name, so the sets can
-# migrate a row at a time. What is no longer name-keyed is IDENTITY: the ranked
+# Rows without an exemplar still decide RELEVANCE by name, which is what let
+# the sets migrate a row at a time. Both shipped sets have now finished --
+# `golden.jsonl` 91/91, `golden-natural.jsonl` 50/50 since #434 -- so NO
+# shipped set exercises this path. It is kept for a set added outside this
+# repo and is covered only by unit tests; treat a coverage report that calls
+# it dead as a statement about the shipped sets, not about the contract.
+# What is no longer name-keyed is IDENTITY: the ranked
 # list de-duplicates on the content hash for every row, exemplar or not. Keying
 # both on the name collapsed 13 different `code-reviewer`s into one rank slot
 # and inflated recall@10 by about one query (0.863 -> 0.852 over the pinned
@@ -366,6 +371,16 @@ def evaluate(rows: list[dict], ranker: Ranker, k: int) -> tuple[list[dict], dict
 
 
 def _aggregate(per_case: list[dict]) -> dict:
+    """Overall metrics, plus an INDICATIVE slice per item kind.
+
+    A row's `kind` is its PRIMARY target's kind, and an exemplar row's
+    relevant set is a content class, which can span kinds: golden.jsonl's
+    Docker and Flutter rows each pin an `agents/*.md` workflow together with
+    a `SKILL.md` that answers the same question, so the `workflow` slice can
+    be credited for retrieving the skill. That is correct grading and a fuzzy
+    slice -- read `by_kind` as "how the gate does on questions shaped like
+    this", not as a per-kind guarantee. Only `overall` is floored.
+    """
     overall = {m: _mean(c["scores"][m] for c in per_case) for m in METRICS}
     by_kind: dict[str, dict] = {}
     for kind in KINDS:

{boost_skill_cli-1.2.153 → boost_skill_cli-1.2.154}/tests/eval/baseline.json

@@ -1,23 +1,5 @@
 {
   "sets": {
-    "golden.jsonl@a0617183f8c9": {
-      "k": 10,
-      "golden": "golden.jsonl",
-      "engines": {
-        "catalog.search": {
-          "recall@k": 0.7307692307692307,
-          "hit@1": 0.46153846153846156,
-          "MRR": 0.561042281984482,
-          "nDCG@k": 0.5927377504239227
-        },
-        "BM25 full-content": {
-          "recall@k": 0.8406593406593407,
-          "hit@1": 0.4835164835164835,
-          "MRR": 0.6065168803770766,
-          "nDCG@k": 0.6552333001662786
-        }
-      }
-    },
     "golden-natural.jsonl@0d91b0cd8e41": {
       "k": 10,
       "golden": "golden-natural.jsonl",
@@ -35,6 +17,24 @@
           "nDCG@k": 0.2591292012176286
         }
       }
+    },
+    "golden.jsonl@d49143dc71a0": {
+      "k": 10,
+      "golden": "golden.jsonl",
+      "engines": {
+        "catalog.search": {
+          "recall@k": 0.7362637362637363,
+          "hit@1": 0.46153846153846156,
+          "MRR": 0.5608885895230973,
+          "nDCG@k": 0.596988880173753
+        },
+        "BM25 full-content": {
+          "recall@k": 0.8461538461538461,
+          "hit@1": 0.4835164835164835,
+          "MRR": 0.6065168803770766,
+          "nDCG@k": 0.6594844299161087
+        }
+      }
     }
   }
 }