boost-skill-cli 1.2.152__tar.gz → 1.2.154__tar.gz

{boost_skill_cli-1.2.152 → boost_skill_cli-1.2.154}/CLAUDE.md

@@ -101,16 +101,23 @@ engines over the golden set (`tests/eval/golden.jsonl`) and floors **four**
 metrics on BM25 — recall@k, hit@1, MRR and nDCG@k. Flooring recall alone was a
 hole rather than a simplification: a ranker that finds the right answer every
 time and never ranks it first scores recall@10 1.000 with hit@1 0.000, and
-passed. The golden set grades real catalog items **by name**, so it needs a
-corpus: `scripts/ensure_eval_corpus.sh` first taps the pinned repo list in
-`tests/eval/taps.txt` (the minimal set covering all golden targets — `boost tap
---defaults` is NOT enough, it omits every rule/workflow repo). The list is
-**twenty** repos: the first six cover every golden target, the rest exist so the
-corpus is a realistic size. That matters more than it sounds — over the six
-alone (921 entries at the current pins) BM25 scores 0.989 / 0.769 / 0.848 /
-0.880, and over the twenty (10,731) it scores **0.841 / 0.484 / 0.607 / 0.655**
-on the same golden set (what `tests/eval/baseline.json` records), so three of
-the four old floors fail once the corpus stops being tiny.
+passed. The golden set grades real catalog items **by the body they name**, so it
+needs a corpus: `scripts/ensure_eval_corpus.sh` first taps the pinned repo list in
+`tests/eval/taps.txt` (`boost tap --defaults` is NOT enough, it omits every
+rule/workflow repo). The list is **twenty** repos and **not one of them
+can be dropped**: the first six are the set the name-graded floors were first
+measured on, and the `# --- scale` divider below them no longer separates
+"targets" from "filler" — `golden.jsonl` pins exemplars into two repos below
+it and `golden-natural.jsonl` into five, so no proper subset of the file can
+score either set. The rest exist so the corpus is a realistic size. That matters more than it sounds — over the six
+alone (921 entries at the current pins) BM25 scored 0.989 / 0.769 / 0.848 /
+0.880, and over the twenty (10,731) it scores **0.846 / 0.484 / 0.607 / 0.659**
+(what `tests/eval/baseline.json` records), so three of the four old floors fail
+once the corpus stops being tiny. The six-repo row is **historical**: it was
+measured name-graded, before the set was pinned, and cannot be re-run against
+today's `golden.jsonl` — see the coupling rule below. It stays re-derivable:
+`git show e11a34f2:tests/eval/golden.jsonl` scored over a six-tap corpus
+reproduces it exactly.
 
 **The ranked list de-duplicates on the content hash, not the name.** A grade key
 decides both relevance and identity, and keying identity on the name collapsed
@@ -118,13 +125,34 @@ decides both relevance and identity, and keying identity on the name collapsed
 ranker with a compression that existed only in the scoring code, and worth about
 one query of recall@10. That is where the old "recall is 1.000" folklore came
 from; the six-repo corpus measured 0.978 at the pins of the time once mirrors
-collapse and homonyms do not. Relevance is still decided by name (or by content
-class when a golden row pins an `exemplar`), so the sets can migrate a row at a
-time.
-
-Each floor sits ~10% under its measured value — loose enough that upstream drift
-can't flake the build, tight enough to catch a collapse. Regression-vs-baseline
-stays relaxed (`--regression-eps 1`), so the absolute floors are the real gate.
+collapse and homonyms do not. Relevance is decided by the content class of the
+bodies a row pins in its `exemplar`, or by name for a row that pins none. The
+two conventions coexist so a set can migrate a row at a time, and **both
+shipped sets have now finished** — `golden.jsonl` 91/91 here, and
+`golden-natural.jsonl` 50/50 back in #434. So no shipped set exercises the
+name path any more: it is kept for a set added outside this repo, and unit
+tests (`TestNameGradingIsUnchanged`) are the only thing covering it. Don't
+delete it on the strength of the coverage report, and don't claim a shipped
+set still uses it.
+
+The floors are fixed absolute values, and the uniform "~10%" margin this file
+claimed from 170d52c0 (2026-07-31) onward was never one of them. They sit under
+their measured values (7.8% / 17.3% / 14.3% / 12.1%), recall@k the tightest —
+loose enough that upstream drift can't flake the build, tight enough to catch a
+collapse, but a 2.21x spread rather than one number, and the tightest of them
+is worth 6.0 queries of 91. The spread is the design working, not drifting: the
+monthly refresh re-measures the row and deliberately leaves the floors where
+they are, so every refresh that improves a metric widens that metric's margin.
+Floors are re-stated after a refresh, never re-derived — a floor that follows
+its own measurement ratchets down with the corpus and ends up measuring the
+corpus. `test_corpus_prose.py` holds every margin above to `baseline.json`, to
+`golden.jsonl` and to the argv of the `Makefile`'s own `eval` recipe — read
+through `eval_retrieval.build_parser()` rather than matched, because the
+tightest floor is `--fail-under` and not a `--floor`, and because a tab-indented
+comment is argv to a regex and a no-op to make. So a refresh that moves the row
+fails the build until someone writes the new margin down and looks at it.
+Regression-vs-baseline stays relaxed (`--regression-eps 1`), so the absolute
+floors are the real gate.
 
 **Every row of `taps.txt` pins a commit SHA**, and `scripts/eval_corpus.py`
 checks each clone out at it — most of the corpus is one third-party repo, so an
@@ -141,6 +169,25 @@ floored recall alone at 0.85 against a measured 0.863 — a buffer of 1.15
 queries out of 91 — and applied none of the other three). Moving a pin means
 regenerating the baseline; the file says how.
 
+**A pinned golden row couples the set to a corpus, and that is the trade.**
+An `exemplar` is `tap::skill_md`, so a row can only be scored over a corpus
+holding that tap: `prepare_row` raises `SystemExit` on a spec it cannot
+resolve, deliberately, because a silent fall back to name grading turns a typo
+into a quietly weaker gate that still reports a number. Consequences:
+
+- **A subset corpus can no longer score `golden.jsonl`.** The six-repo
+  comparison figure is historical for this reason, not stale. `taps-scale.txt`
+  is fine — it holds every row of `taps.txt` and 129 more, so
+  `eval-scale.yml` keeps scoring the same set.
+- **A pin move can invalidate a path.** `eval-corpus-refresh.yml`'s re-baseline
+  step is not `continue-on-error`, so a renamed file stops the monthly refresh
+  before its pull request opens. That is the intended signal and the fix is to
+  re-resolve the path in the new clone — `--worksheet` will not help, because
+  it lists rows still graded **by name** and a pinned row is not one.
+- **Do not "fix" it by tolerating an unresolvable spec in a multi-spec list.**
+  A refresh that drops a repo would then narrow every such row's class to
+  whatever still resolves, which is the same silent weakening one level down.
+
 **Baselines are keyed by query set** (`name@content-digest`), so one file holds
 both `golden.jsonl` and `golden-natural.jsonl` without either overwriting the
 other. Before that, running the natural-language set printed eight confident

{boost_skill_cli-1.2.152 → boost_skill_cli-1.2.154}/Makefile

@@ -122,14 +122,24 @@ dist-check:
 # 0.000 passed.
 #
 # The floors are calibrated against a TWENTY-tap corpus, not the six it used to
-# be. Over the six (921 entries) BM25 scores 0.989 / 0.769 / 0.848 / 0.880;
-# over twenty it scores 0.841 / 0.484 / 0.607 / 0.655 on the same golden set,
-# which is what tests/eval/baseline.json records at the current pins. The
+# be. Over the six (921 entries) BM25 scored 0.989 / 0.769 / 0.848 / 0.880;
+# over twenty it scores 0.846 / 0.484 / 0.607 / 0.659, which is what
+# tests/eval/baseline.json records at the current pins. Not the same golden
+# set: the six-repo row is name-graded and HISTORICAL, measured before every
+# row pinned an exemplar, and today's golden.jsonl cannot be scored over six
+# taps at all (a pin naming a tap outside the corpus is a hard error). It is
+# still re-derivable -- `git show e11a34f2:tests/eval/golden.jsonl` over a
+# six-tap corpus reproduces it to three places. The
 # second set is what a real user sees, so flooring against the first was
 # measuring the corpus rather than the retrieval — three of the four old floors
-# fail outright once the corpus is realistic. Each floor now sits ~10% below
-# its measured value: loose enough that a query or two cannot flake the build,
-# tight enough that a collapse fails it. Regression-vs-baseline stays relaxed
+# fail outright once the corpus is realistic. The four floors sit a margin
+# under their measured values (7.8% / 17.3% / 14.3% / 12.1%), recall@k the
+# tightest at 6.0 queries of 91: loose enough that a query or two cannot flake
+# the build, tight enough that a collapse fails it. That margin is not uniform
+# and never was -- it is a 2.21x spread, because the row moves with every
+# monthly refresh and the floors deliberately do not. So it is quoted rather
+# than asserted, and tests/unit/test_corpus_prose.py fails until the new
+# spread is written here. Regression-vs-baseline stays relaxed
 # (--regression-eps 1); the absolute floors are the real gate. (It was relaxed
 # while the corpus tracked upstream HEAD. tests/eval/taps.txt pins a commit per
 # row now, so that is no longer the reason.)

{boost_skill_cli-1.2.152 → boost_skill_cli-1.2.154}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.2.152
+Version: 1.2.154
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License-Expression: Apache-2.0

{boost_skill_cli-1.2.152 → boost_skill_cli-1.2.154}/boost_cli/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '1.2.152'
-__version_tuple__ = version_tuple = (1, 2, 152)
+__version__ = version = '1.2.154'
+__version_tuple__ = version_tuple = (1, 2, 154)
 
 __commit_id__ = commit_id = None

{boost_skill_cli-1.2.152 → boost_skill_cli-1.2.154}/boost_skill_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.2.152
+Version: 1.2.154
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License-Expression: Apache-2.0

{boost_skill_cli-1.2.152 → boost_skill_cli-1.2.154}/docs/eval.html

@@ -146,7 +146,7 @@
 
     <div class="stats">
       <div class="stat reveal"><b>5</b><span>Evaluation tiers</span></div>
-      <div class="stat reveal"><b>0.841</b><span>BM25 recall@10 · pinned corpus</span></div>
+      <div class="stat reveal"><b>0.846</b><span>BM25 recall@10 · pinned corpus</span></div>
       <div class="stat reveal"><b>141</b><span>Golden judgments</span></div>
       <div class="stat reveal"><b>0</b><span>Deps in the required gate</span></div>
     </div>
@@ -326,7 +326,8 @@
         regression &mdash; it is what users already had, now visible.</p>
 
       <h3>2 &middot; The score was also about the shape of the question</h3>
-      <p>The keyword golden set grades catalog items <b>by name</b>, so its queries carry the
+      <p>The keyword golden set was graded <b>by name</b> when these runs were made &mdash; both
+        shipped sets now pin an exemplar body on every row &mdash; and its queries carry the
         vocabulary of the thing they are looking for. That is BM25's home ground. The 50 queries in
         <code>golden-natural.jsonl</code> describe a <em>problem</em> instead &mdash; each written
         from its target's own description, with the target's distinctive name tokens deliberately
@@ -341,7 +342,8 @@
         query sets, <code>k=10</code>. <b>Voyage is absent: it needs an API key, and these runs were
         keyless</b>, which is the configuration most users are in.</p>
 
-      <p><b>Keyword golden set</b> &mdash; 91 queries that grade items by name:</p>
+      <p><b>Keyword golden set</b> &mdash; 91 queries, graded by name at the pins these runs
+        used; every row pins an exemplar body today:</p>
       <table class="contract">
         <thead><tr><th>engine</th><th>recall@10</th><th>hit@1</th><th>MRR</th><th>nDCG@10</th></tr></thead>
         <tbody>
@@ -397,8 +399,10 @@
     <div class="wrap">
       <span class="eyebrow">Sequence · the required gate</span>
       <h2 class="sec-title">How Tier 1 runs on every PR</h2>
-      <p class="lead" style="margin-top:20px">The golden set grades items <em>by name</em>, so the
-        repos that hold them must be present. CI taps a pinned corpus first, builds the BM25 index,
+      <p class="lead" style="margin-top:20px">Every golden row pins the <em>body</em> it was
+        written about, as <code>tap::skill_md</code>, so the repos that hold them must be present
+        &mdash; a pin the corpus cannot resolve halts the gate and names the missing spec,
+        rather than quietly scoring a smaller set. CI taps a pinned corpus first, builds the BM25 index,
         scores the golden set, and floors recall — all offline after the tap, all deterministic.</p>
 
       <div class="diagram">

{boost_skill_cli-1.2.152 → boost_skill_cli-1.2.154}/docs/roadmap/items/eval-floor-calibration-stale-after-pin-refresh.md

@@ -1,16 +1,16 @@
 ---
 id: eval-floor-calibration-stale-after-pin-refresh
 board: code
-section: planned
-status: planned
+section: shipped
+status: shipped
 category: Quality · Retrieval eval
 complexity: M
 impact: Med
 wow: 3
 note: CLAUDE.md states the gate's four floors sit "~10% under" measured values of 0.852 / 0…
 order: 210
-owner:
-pr:
+owner: loop/eval-floor-calibration
+pr: "993"
 title: The 2026-09-01 pin refresh moved the corpus and re-baselined it, but nothing re-derives the floors — CLAUDE.md's "~10% under measured" is now 7.2%–17.3%
 ---
 <b>Measured.</b> The required gate's recall floor has 5.52 queries of headroom out of 91 ((0.8407 - 0.78) x 91), against the 6.55 that CLAUDE.md's published 0.852 implies — one golden query of margin the documentation says exists and does not, because commit cbc0a58b re-baselined the corpus to 10,731 entries and left CLAUDE.md, taps.txt and eval_corpus.py all stating 10,152 / 0.852.
@@ -32,7 +32,7 @@ title: The 2026-09-01 pin refresh moved the corpus and re-baselined it, but noth
 
 <b>What adversarial verification corrected.</b> This card's numbers are the re-measured ones, not the ones first reported.
 
-1. CAUSAL FRAMING IS WRONG (the headline correction). The title and claim say the 2026-09-01 pin refresh is what made "~10% under" untrue. It was untrue the day it was written. <code>git log -S'~10% under'</code> traces the sentence to 170d52c0 (2026-07-31), whose own commit message says the floors were "re-derived against the 20-tap numbers at the same ~10% relative headroom" against measured 0.863 / 0.473 / 0.607 / 0.662 — at which point the gaps were 9.6% / 15.4% / 14.3% / 12.4%, a 1.60x spread. The refresh WIDENED a spread (1.60x -&gt; 1.82x -&gt; 2.39x); it did not create one. Correct statement: "the floors were never uniformly ~10% under, and the refresh widened the spread from 1.82x to 2.39x."
+1. CAUSAL FRAMING IS WRONG (the headline correction). The title and claim say the 2026-09-01 pin refresh is what made "~10% under" untrue. It was untrue the day it was written. <code>git log -S'~10% under'</code> traces the sentence to 170d52c0 (2026-07-31), whose own commit message says the floors were "re-derived against the 20-tap numbers at the same ~10% relative headroom" against measured 0.863 / 0.473 / 0.607 / 0.662 — at which point the gaps were 9.6% / 15.3% / 14.4% / 12.4%, a 1.60x spread. (This line first read 9.6% / 15.4% / 14.3% / 12.4%; recomputing from 170d52c0's own <code>baseline.json</code> and <code>Makefile</code> gives 9.5796 / 15.3488 / 14.3831 / 12.4408, so two digits were transposed.) The spread widened twice and by two different mechanisms, only one of which is a refresh: 1.60x at 170d52c0, 1.82x at d8da2e71 — <em>de-duplicate the ranked list by body, not by name</em>, a ranker correctness fix (#411) — and 2.39x at the 2026-09-01 refresh cbc0a58b. Correct statement: "the floors were never uniformly ~10% under, and nothing since has narrowed the spread."
 
 2. "not the 6.60 the documented number implies" — 6.60 matches nothing. CLAUDE.md's published 0.852 gives (0.852-0.78) x 91 = 6.55 queries; the unrounded pre-refresh baseline 0.8406... gives 6.52. Use 6.55. (The optimism is therefore ~1.03 queries, not 1.1.)
 
@@ -53,3 +53,9 @@ SCOPE LIMIT #2: the shared $TMPDIR/eval-home is materialised at the PRE-refresh
 <b>Why it is worth doing.</b> The "~10% under" statement is the entire published justification for where the four floors sit — CLAUDE.md calls it "loose enough that upstream drift can't flake the build, tight enough to catch a collapse." That is a claim about a margin, and the margin now differs 2.4x across the four metrics with recall the tightest at 5.52 queries out of 91. Because the monthly refresh re-baselines but never re-derives or re-states the floors, and prompts a human only when the refreshed corpus FAILS, every future passing refresh decays the calibration further with no signal at all.
 
 <em>Found by an automated audit of retrieval/eval quality, the search &amp; browse surfaces, and first-run onboarding; every finding was then re-measured from scratch by an independent adversarial verifier whose instruction was to refute it. Verdict: <b>CORRECTED</b>. No fix is prescribed here — the measurement is the contribution.</em>
+
+<b>What the fix turned out to be, and what the card got wrong about scope.</b> Half of this card was already fixed when it was picked up. The figure half — "CLAUDE.md, taps.txt and eval_corpus.py all stating 10,152 / 0.852" — is stale: <code>tests/unit/test_corpus_prose.py</code> now holds every one of those figures to <code>taps.txt</code>'s own rows and to <code>baseline.json</code>, so CLAUDE.md reads 10,731 and 0.841 / 0.484 / 0.607 / 0.655 and cannot drift again. The card's line references (CLAUDE.md:67,84) no longer point at anything. What survived is the card's own stated core, and it is the harder half: the <em>characterisation</em> "~10% under" was prose, was false the day it was written, and no mechanism touched it.
+
+<b>The remedy is the card's own, not the one the finding's wording invites.</b> No floor moves and no headroom gate is added — a floor that follows its own measurement ends up measuring the corpus, which is exactly what 170d52c0 warned about. Instead the true margin is written down at measured precision (7.2% / 17.3% / 14.3% / 11.5%, a 2.39x spread, recall@k binding at 5.5 queries of 91) and held by the <code>QUOTED</code> mechanism that already holds the figures beside it. A refresh that moves the row now fails the build until a human restates the margin and looks at it — which is the signal the card says is missing, without touching the gate it says not to touch.
+
+<b>Why the required set drifted and the advisory set did not.</b> The contrast is the tell. <code>make eval-natural</code>'s floors really are ~10% under (11.1 / 12.5 / 11.4 / 11.2, a 1.12x spread) because <code>tests/unit/test_eval_corpus.py</code> moves them with the row in both directions. The required set, where the claim actually matters, had no such mechanism — so the one place the margin was load-bearing is the one place nothing checked it.

{boost_skill_cli-1.2.152 → boost_skill_cli-1.2.154}/docs/roadmap/items/eval-gate-scores-a-body-less-corpus.md

@@ -52,6 +52,6 @@ WHY Med, NOT High — two facts I verified that cap the blast radius: 1. CI cann
 
 SCOPE LIMITS OF MY REPRO — a card author must not blur these: - I measured the FULLY-missing case. The "half-materialised tap tree" phrasing in <code>why_it_matters</code> was NOT measured by me; degradation is presumably proportional but is unverified.
 
-<b>Why it is worth doing.</b> The gate whose whole job is to say "retrieval still works over a real corpus" returns a confident, better-than-baseline PASS over a corpus that is name+description only — 95% of its text absent. A developer who reclaims the 265 MB <code>repos/</code> tree, or whose tap tree is half-materialised, gets four green floors that attest to nothing, and the greener number makes the loss look like an improvement rather than a defect. Every downstream floor calibration ("each floor sits ~10% under its measured value") is stated against 0.473/0.605/0.657, which the body-less run clears by 48%/33%/25% instead.
+<b>Why it is worth doing.</b> The gate whose whole job is to say "retrieval still works over a real corpus" returns a confident, better-than-baseline PASS over a corpus that is name+description only — 95% of its text absent. A developer who reclaims the 265 MB <code>repos/</code> tree, or whose tap tree is half-materialised, gets four green floors that attest to nothing, and the greener number makes the loss look like an improvement rather than a defect. Every downstream floor calibration is stated against 0.473/0.605/0.657, which the body-less run clears by 48%/33%/25% instead. (This line quoted CLAUDE.md's "each floor sits ~10% under its measured value" as the live convention. That sentence was retired in PR #993: the margins were never uniform — 7.2% / 17.3% / 14.3% / 11.5% at the current pins — and CLAUDE.md now quotes all four and holds them.)
 
 <em>Found by an automated audit of retrieval/eval quality, the search &amp; browse surfaces, and first-run onboarding; every finding was then re-measured from scratch by an independent adversarial verifier whose instruction was to refute it. Verdict: <b>CORRECTED</b>. No fix is prescribed here — the measurement is the contribution.</em>

{boost_skill_cli-1.2.152 → boost_skill_cli-1.2.154}/docs/roadmap/items/gated-golden-set-has-zero-exemplars.md

@@ -1,17 +1,17 @@
 ---
 id: gated-golden-set-has-zero-exemplars
 board: code
-section: planned
-status: planned
+section: shipped
+status: shipped
 category: Quality · Retrieval eval
 complexity: M
 impact: Med
 wow: 3
 note: CLAUDE.md presents exemplar pinning as a live migration — "relevance is still decided…
 order: 214
-owner:
-pr:
-title: The exemplar mechanism was applied to the ungated set only: golden.jsonl is 0/91 pinned, and 10 of its 43 hit@1 credits are on names the metric cannot adjudicate
+owner: loop/gated-golden-exemplars
+pr: "994"
+title: The exemplar mechanism was applied to the ungated set only: golden.jsonl is 0/91 pinned, and 10 of its 44 hit@1 credits are on names the metric cannot adjudicate
 ---
 <b>Measured.</b> On the 10,731-entry corpus that committed <code>taps.txt</code> materializes today, <code>tests/eval/golden.jsonl</code> — the only set the required <code>eval</code> gate floors — is 0/91 exemplar-pinned while the ungated <code>golden-natural.jsonl</code> is 50/50; 10 of its 44 hit@1 credits (22.7%) are awarded on a name that resolves to more than one distinct body, and the hit@1 floor's entire headroom is 7.60 queries, smaller than the 10 credits the metric cannot adjudicate.
 
@@ -19,7 +19,7 @@ title: The exemplar mechanism was applied to the ungated set only: golden.jsonl
 
 <code>cd &lt;repo&gt;</code><br>
 <code>export BOOST_HOME=$TMPDIR/eval-home</code><br>
-<code># 0. confirm which corpus you are on (expect 10152)</code><br>
+<code># 0. confirm which corpus you are on (expect 10731)</code><br>
 <code>.venv/bin/python -c "import sys;sys.path.insert(0,'.');from boost_cli.core import catalog;print(len(catalog.all_entries()))"; echo EXIT=$?</code><br>
 <code># 1. exemplar counts per set</code><br>
 <code>python3 -c "</code><br>
@@ -52,6 +52,16 @@ Nothing else in the finding is wrong. 0/91 vs 50/50, the 27 undecided rows / 62
 
 5. NOT A DEFECT IN THE HARNESS: <code>exemplar_worksheet</code> (scripts/eval_retrieval.py:219) and exemplar grading work correctly and fail loudly on a bad pin.
 
-<b>Why it is worth doing.</b> golden.jsonl is the set the required <code>eval</code> gate floors, so it is the only one that can block a merge. 23.3% of its hit@1 credits are awarded on a name that maps to several genuinely different skills, and the floor's whole margin (6.60 queries) is narrower than the un-adjudicated credit count (10). That does not mean retrieval is worse than reported — the realistic shift is 1-2 queries — it means the published margin cannot be read as precision about the intended skill, so anyone tuning blend weights or pool depth against this gate inherits an unquantified slack.
+<b>Why it is worth doing.</b> golden.jsonl is the set the required <code>eval</code> gate floors, so it is the only one that can block a merge. 22.7% of its hit@1 credits are awarded on a name that maps to several genuinely different skills, and the floor's whole margin (7.60 queries) is narrower than the un-adjudicated credit count (10). That does not mean retrieval is worse than reported — the realistic shift is 1-2 queries — it means the published margin cannot be read as precision about the intended skill, so anyone tuning blend weights or pool depth against this gate inherits an unquantified slack.
 
 <em>Found by an automated audit of retrieval/eval quality, the search &amp; browse surfaces, and first-run onboarding; every finding was then re-measured from scratch by an independent adversarial verifier whose instruction was to refute it. Verdict: <b>CORRECTED</b>. No fix is prescribed here — the measurement is the contribution.</em>
+
+<b>Shipped.</b> All 91 rows of <code>tests/eval/golden.jsonl</code> now pin an <code>exemplar</code>, so the required gate grades by content class rather than by name. <code>--worksheet</code> reports 0 rows still name-graded, and <code>prepare_row</code> raises on a spec that resolves to nothing, so a stale pin fails the gate rather than quietly weakening it.
+
+<b>The pins are a judgment, so they were made as one.</b> 64 of the 91 rows resolved to a single body and were pinned by lookup. Clustering the remaining 27 rows' candidate bodies mechanically (difflib ratio 0.90) collapsed 7 of them to a single re-publication cluster, pinnable as a list by <code>diff</code> rather than opinion. That left <b>20</b> genuine homonym rows, each judged by an agent under one rule — keep a cluster if a user typing this query would be correctly served by it; two independent implementations of the same job both count; drop only a cluster doing a different job — and each judgment then attacked by two adversarial verifiers, one hunting over-narrow pins and one hunting over-broad. 60 agents, <b>0 of 20 decisions refuted</b>, 19 of 20 at high confidence. Two clusters were dropped: Sentry's <code>brand-guidelines</code> (copy-writing, not a visual brand system) and oh-my-agent's <code>pdf</code> (PDF&rarr;Markdown conversion, which cannot fill a form field).
+
+<b>What the measurement did, and did not, do.</b> Scored per query against the old set and the new on the same 10,731-entry corpus, <b>exactly 1 of 91 rows changed</b>. The other 90 grade identically under name and content class, which is the honest result: on this corpus the name was already selecting the right body almost everywhere, and the value of pinning is that it can no longer stop doing so silently. The one mover is the set's only two-name row (<code>style an html artifact with a color and font theme</code>, relevant <code>theme-factory</code> + <code>web-artifacts-builder</code>): <code>relevant</code> is documented as names that <em>count as a correct top hit</em> — alternatives — but <code>recall@k</code> over a two-element set scored 0.5 for finding one of them. The four pinned bodies are one class, so it scores 1.0, and the aggregate moved 76.5/91 &rarr; 77.0/91. <code>hit@1</code> and <code>MRR</code> are bit-identical.
+
+BM25 over the twenty taps: 0.841 / 0.484 / 0.607 / 0.655 &rarr; <b>0.846 / 0.484 / 0.607 / 0.659</b>. All four floors still clear, and none was moved &mdash; margins are now 7.8% / 17.3% / 14.3% / 12.1%, a 2.21x spread, recall@k binding at 6.0 queries of 91.
+
+<b>The cost, recorded rather than discovered later.</b> A pin is <code>tap::skill_md</code>, so the set can only be scored over a corpus holding that tap. The six-repo comparison figure in <code>Makefile</code> is therefore <b>historical</b>, not stale: today's <code>golden.jsonl</code> cannot be scored over six taps at all. It stays re-derivable &mdash; <code>git show e11a34f2:tests/eval/golden.jsonl</code> over a six-tap corpus reproduces 0.989 / 0.769 / 0.848 / 0.880 to three places. <code>taps-scale.txt</code> is unaffected: it is a superset of <code>taps.txt</code> (149 rows &supe; 20), so <code>eval-scale.yml</code> keeps scoring the same set. The real exposure is <code>eval-corpus-refresh.yml</code>, whose re-baseline step is not <code>continue-on-error</code>: a monthly pin move past a renamed file now stops 141 pinned rows instead of 50, and the pull request does not open. That is the intended signal and is written up as a non-obvious rule in <code>CLAUDE.md</code>.

{boost_skill_cli-1.2.152 → boost_skill_cli-1.2.154}/docs/roadmap/items/required-eval-gate-still-grades-by-name.md

@@ -1,16 +1,16 @@
 ---
 id: required-eval-gate-still-grades-by-name
 board: code
-section: planned
-status: planned
+section: shipped
+status: shipped
 category: Tech-debt
 complexity: M
 impact: Med
 wow: 3
 note: CLAUDE.md states "Relevance is still decided by name (or by content class when a gold…
 order: 243
-owner:
-pr:
+owner: loop/gated-golden-exemplars
+pr: "994"
 title: The exemplar migration never reached the query set the required gate runs: 0 of 91 rows, and 27 of them have an ambiguous target on the gate's own corpus
 ---
 <b>Measured.</b> Both invocations of the merge-blocking retrieval gate — the Makefile <code>eval</code> target and ci.yml's "retrieval quality gate" step — run <code>scripts/eval_retrieval.py</code> with no <code>--golden</code>, so both take <code>DEFAULT_GOLDEN = tests/eval/golden.jsonl</code>; 0 of that file's 91 rows carry an <code>exemplar</code>, and on the gate's own 20-tap corpus 27 of the 91 rows have at least one target name that resolves to more than one content digest (21 of 77 distinct target names are ambiguous).
@@ -40,3 +40,11 @@ Floor-margin exposure, with that qualification attached: hit@1 measures 44/91 =
 <b>Why it is worth doing.</b> This is the only gate in <code>make check</code> that can fail a merge on retrieval quality, and every retrieval decision validated against it (RRF fusion vs preferring dense, pool depth, whether the LLM rerank earns its keep) inherits the looseness. The project already established the fix and built the harness for it; the required set simply never got migrated, and because the card is closed as shipped nothing tracks the remaining work on the set that matters most.
 
 <em>Found by an automated audit of retrieval/eval quality, the search &amp; browse surfaces, and first-run onboarding; every finding was then re-measured from scratch by an independent adversarial verifier whose instruction was to refute it. Verdict: <b>CONFIRMED</b>. No fix is prescribed here — the measurement is the contribution.</em>
+
+<b>Shipped, with the card above.</b> This card and <code>gated-golden-set-has-zero-exemplars</code> measured one defect from two angles — 0/91 pinned rows, and 27 of those rows ambiguous on the gate's own corpus — and one change closes both. All 91 rows of <code>tests/eval/golden.jsonl</code> now pin an <code>exemplar</code>; the required gate grades by content class.
+
+<b>This card's severity call was right, and the measurement confirms it.</b> The verifier lowered severity from High to Medium on the grounds that most ambiguous names are re-publications of one upstream skill, so a "wrong body" was still a correct answer. That is exactly what the pins found: the 20 judged rows kept 2-4 bodies each and dropped only <b>2</b> clusters in total, and scoring per query showed <b>90 of 91 rows grade identically</b> under name and content class. The looseness was real and worth closing, and it was not inflating the published numbers.
+
+The card's own row arithmetic predicted the result before the pins existed: it stated <em>"recall@10 measures 77/91 = 0.846 against a floor needing 71 rows (margin 6 rows)"</em>, counting a row as matched or not. The metric averaged 76.5/91 = 0.841 because the set's one two-name row scored 0.5 for finding one of two documented alternatives. Pinning makes those alternatives one class, so the metric now reads 77/91 = <b>0.846</b> — the card's figure, arrived at independently.
+
+<b>What was NOT done, deliberately.</b> No floor was moved. The four floors stay at 0.78 / 0.40 / 0.52 / 0.58 and all four still clear; #993 established in this same file that a floor is restated after a measurement, never re-derived from it. The name-grading path is kept, not deleted &mdash; but nothing shipped exercises it any more: <code>golden-natural.jsonl</code> finished migrating in #434 and is 50/50, so with golden.jsonl at 91/91 both shipped sets are fully pinned. It stays for a query set added outside this repo, covered by <code>TestNameGradingIsUnchanged</code> alone.

{boost_skill_cli-1.2.152 → boost_skill_cli-1.2.154}/docs/roadmap.html

@@ -140,9 +140,9 @@
   <section class="block reveal">
     <div class="stats">
       <!-- ROADMAP:stats -->
-      <div class="stat"><b>24</b><span>Shipped</span></div>
+      <div class="stat"><b>27</b><span>Shipped</span></div>
       <div class="stat"><b>1</b><span>Next up</span></div>
-      <div class="stat"><b>77</b><span>Planned</span></div>
+      <div class="stat"><b>74</b><span>Planned</span></div>
       <div class="stat"><b>505</b><span>Loop finds</span></div>
       <!-- /ROADMAP:stats -->
     </div>
@@ -420,6 +420,42 @@
         </div>
       </article>
 
+      <article class="cap rcard" id="eval-floor-calibration-stale-after-pin-refresh">
+        <div class="head"><span class="pill shipped">Shipped</span><span class="cat">Quality · Retrieval eval</span></div>
+        <h3>The 2026-09-01 pin refresh moved the corpus and re-baselined it, but nothing re-derives the floors — CLAUDE.md's "~10% under measured" is now 7.2%–17.3%</h3>
+        <p class="cardbody"><a class="writeup" href="https://github.com/jonnyeclectic/boost/blob/main/docs/roadmap/items/eval-floor-calibration-stale-after-pin-refresh.md">Write-up &middot; eval-floor-calibration-stale-after-pin-refresh.md</a></p>
+        <div class="meta">
+          <span class="m">Complexity <b>M</b></span>
+          <span class="m">Impact <b>Med</b></span>
+          <span class="m">Wow <b class="wow">★★★</b></span>
+          <span class="m">CLAUDE.md states the gate's four floors sit "~10% under" measured values of 0.852 / 0…</span>
+        </div>
+      </article>
+
+      <article class="cap rcard" id="gated-golden-set-has-zero-exemplars">
+        <div class="head"><span class="pill shipped">Shipped</span><span class="cat">Quality · Retrieval eval</span></div>
+        <h3>The exemplar mechanism was applied to the ungated set only: golden.jsonl is 0/91 pinned, and 10 of its 44 hit@1 credits are on names the metric cannot adjudicate</h3>
+        <p class="cardbody"><a class="writeup" href="https://github.com/jonnyeclectic/boost/blob/main/docs/roadmap/items/gated-golden-set-has-zero-exemplars.md">Write-up &middot; gated-golden-set-has-zero-exemplars.md</a></p>
+        <div class="meta">
+          <span class="m">Complexity <b>M</b></span>
+          <span class="m">Impact <b>Med</b></span>
+          <span class="m">Wow <b class="wow">★★★</b></span>
+          <span class="m">CLAUDE.md presents exemplar pinning as a live migration — "relevance is still decided…</span>
+        </div>
+      </article>
+
+      <article class="cap rcard" id="required-eval-gate-still-grades-by-name">
+        <div class="head"><span class="pill shipped">Shipped</span><span class="cat">Tech-debt</span></div>
+        <h3>The exemplar migration never reached the query set the required gate runs: 0 of 91 rows, and 27 of them have an ambiguous target on the gate's own corpus</h3>
+        <p class="cardbody"><a class="writeup" href="https://github.com/jonnyeclectic/boost/blob/main/docs/roadmap/items/required-eval-gate-still-grades-by-name.md">Write-up &middot; required-eval-gate-still-grades-by-name.md</a></p>
+        <div class="meta">
+          <span class="m">Complexity <b>M</b></span>
+          <span class="m">Impact <b>Med</b></span>
+          <span class="m">Wow <b class="wow">★★★</b></span>
+          <span class="m">CLAUDE.md states "Relevance is still decided by name (or by content class when a gold…</span>
+        </div>
+      </article>
+
       <article class="cap rcard" id="install-cannot-tell-a-skill-needs-another-one">
         <div class="head"><span class="pill shipped">Shipped</span><span class="cat">Install · Research</span></div>
         <h3>Research: how <code>boost install</code> could tell that a skill needs something else installed first</h3>
@@ -1208,57 +1244,6 @@ NOT ALREADY CARDED, but read the existing card first.
         </div>
       </article>
 
-      <article class="cap rcard" id="eval-floor-calibration-stale-after-pin-refresh">
-        <div class="head"><span class="pill planned">Planned</span><span class="cat">Quality · Retrieval eval</span></div>
-        <h3>The 2026-09-01 pin refresh moved the corpus and re-baselined it, but nothing re-derives the floors — CLAUDE.md's "~10% under measured" is now 7.2%–17.3%</h3>
-        <p><b>Measured.</b> The required gate's recall floor has 5.52 queries of headroom out of 91 ((0.8407 - 0.78) x 91), against the 6.55 that CLAUDE.md's published 0.852 implies — one golden query of margin the documentation says exists and does not, because commit cbc0a58b re-baselined the corpus to 10,731 entries and left CLAUDE.md, taps.txt and eval_corpus.py all stating 10,152 / 0.852.
-
-<b>Reproduce it.</b>
-
-<code>cd &lt;repo&gt;</code><br>
-<code># 1. the refresh commit moved pins + baseline and nothing else</code><br>
-<code>git show cbc0a58b --stat --format='%h %ad %s' --date=short</code><br>
-<code>git show cbc0a58b -- tests/eval/baseline.json | grep -E '^[-+].*(recall|hit@1|MRR|nDCG)'</code><br>
-<code># 2. what taps.txt records NOW vs what the docs claim</code><br>
-<code>awk '!/^#/ &amp;&amp; NF&gt;=3 {s+=$3; n++} END {print "rows:", n, " recorded total entries:", s}' tests/eval/taps.txt</code><br>
-<code>grep -n '10,152\|0\.852' CLAUDE.md tests/eval/taps.txt scripts/eval_corpus.py</code><br>
-<code># 3. recompute the floor gaps against the committed post-refresh baseline</code><br>
-<code>python3 -c "</code><br>
-<code>import json</code><br>
-<code>b=json.load(open('tests/eval/baseline.json'))['sets']['golden.jsonl@a0617183f8c9']['engines']['BM25 full-content']</code><br>
-…
-
-<b>What adversarial verification corrected.</b> This card's numbers are the re-measured ones, not the ones first reported.
-
-1. CAUSAL FRAMING IS WRONG (the headline correction). The title and claim say the 2026-09-01 pin refresh is what made "~10% under" untrue. It was untrue the day it was written. <code>git log -S'~10% under'</code> traces the sentence to 170d52c0 (2026-07-31), whose own commit message says the floors were "re-derived against the 20-tap numbers at the same ~10% relative headroom" against measured 0.863 / 0.473 / 0.607 / 0.662 — at which point the gaps were 9.6% / 15.4% / 14.3% / 12.4%, a 1.60x spread. The refresh WIDENED a spread (1.60x -&gt; 1.82x -&gt; 2.39x); it did not create one. Correct statement: "the floors were never uniformly ~10% under, and the refresh widened the spread from 1.82x to 2.39x."
-
-2. "not the 6.60 the documented number implies" — 6.60 matches nothing. CLAUDE.md's published 0.852 gives (0.852-0.78) x 91 = 6.55 queries; the unrounded pre-refresh baseline 0.8406... gives 6.52. Use 6.55. (The optimism is therefore ~1.03 queries, not 1.1.)
-
-3. "2.4x spread" -&gt; 2.39x (rounding, harmless, but state it as 2.39x on a card).
-
-4. DIRECTION OF DRIFT IS NOT UNIFORM. "every future passing refresh decays the calibration further" is true only of the SPREAD. This refresh moved two floors tighter (recall 8.4%-&gt;7.2%, nDCG 11.8%-&gt;11.5%) and two LOOSER (hit@1 15.3%-&gt;17.3%, MRR 14.0%-&gt;14.3%). A card must not say "the floors tightened."
-
-5.
-
-<b>What is NOT established.</b> Recorded because a card that overstates its own evidence is worse than no card.
-
-SCOPE THE REMEDY CAREFULLY — the finding's phrase "nothing re-derives the floors" invites exactly the fix 170d52c0 warns against: auto-moving floors on every refresh ("Lowering a threshold deserves suspicion... a floor calibrated on an unrepresentative corpus measures the corpus"). The repo's design is deliberately fixed absolute floors with a relaxed regression-vs-baseline. The CONFIRMED defect is narrower and should be carded as such: the measured values and the "~10%" characterisation are stated in seven places across four files (CLAUDE.md:67,84; tests/eval/taps.txt:25,43; scripts/eval_corpus.py:30,108 and the runtime-printed operator message at :331-334), nothing re-states or guards them after a PASSING refresh, and no test asserts any of them.
-
-SCOPE LIMIT OF MY REPRO #1: I did not re-measure 0.8407 / 0.4835 / 0.6065 / 0.6552. Those are the committed baseline written by cbc0a58b — which is also the finder's only source. Verifying them empirically needs a re-tap at the new pins, which is forbidden here (shared read-only corpus) and needs network. Every derived percentage in this verification inherits that.
-
-SCOPE LIMIT #2: the shared $TMPDIR/eval-home is materialised at the PRE-refresh pins — live total 10,152 vs taps.txt's recorded 10,731, with the same 10 taps PIN-DIFFERS. <code>make eval</code> against it would trip eval_corpus.py's CORPUS DRIFT check, not run the gate.
-
-<b>Why it is worth doing.</b> The "~10% under" statement is the entire published justification for where the four floors sit — CLAUDE.md calls it "loose enough that upstream drift can't flake the build, tight enough to catch a collapse." That is a claim about a margin, and the margin now differs 2.4x across the four metrics with recall the tightest at 5.52 queries out of 91. Because the monthly refresh re-baselines but never re-derives or re-states the floors, and prompts a human only when the refreshed corpus FAILS, every future passing refresh decays the calibration further with no signal at all.
-
-<em>Found by an automated audit of retrieval/eval quality, the search &amp; browse surfaces, and first-run onboarding; every finding was then re-measured from scratch by an independent adversarial verifier whose instruction was to refute it. Verdict: <b>CORRECTED</b>. No fix is prescribed here — the measurement is the contribution.</em></p>
-        <div class="meta">
-          <span class="m">Complexity <b>M</b></span>
-          <span class="m">Impact <b>Med</b></span>
-          <span class="m">Wow <b class="wow">★★★</b></span>
-          <span class="m">CLAUDE.md states the gate's four floors sit "~10% under" measured values of 0.852 / 0…</span>
-        </div>
-      </article>
-
       <article class="cap rcard" id="eval-gate-scores-a-body-less-corpus">
         <div class="head"><span class="pill shipped">Shipped</span><span class="cat">Quality · Retrieval eval</span></div>
         <h3><code>make eval</code> scores a corpus with every SKILL.md body missing, reports all four floors PASS, and scores HIGHER than the real corpus</h3>
@@ -1295,59 +1280,6 @@ SCOPE LIMIT #2: the shared $TMPDIR/eval-home is materialised at the PRE-refresh
         </div>
       </article>
 
-      <article class="cap rcard" id="gated-golden-set-has-zero-exemplars">
-        <div class="head"><span class="pill planned">Planned</span><span class="cat">Quality · Retrieval eval</span></div>
-        <h3>The exemplar mechanism was applied to the ungated set only: golden.jsonl is 0/91 pinned, and 10 of its 43 hit@1 credits are on names the metric cannot adjudicate</h3>
-        <p><b>Measured.</b> On the 10,731-entry corpus that committed <code>taps.txt</code> materializes today, <code>tests/eval/golden.jsonl</code> — the only set the required <code>eval</code> gate floors — is 0/91 exemplar-pinned while the ungated <code>golden-natural.jsonl</code> is 50/50; 10 of its 44 hit@1 credits (22.7%) are awarded on a name that resolves to more than one distinct body, and the hit@1 floor's entire headroom is 7.60 queries, smaller than the 10 credits the metric cannot adjudicate.
-
-<b>Reproduce it.</b>
-
-<code>cd &lt;repo&gt;</code><br>
-<code>export BOOST_HOME=$TMPDIR/eval-home</code><br>
-<code># 0. confirm which corpus you are on (expect 10152)</code><br>
-<code>.venv/bin/python -c "import sys;sys.path.insert(0,'.');from boost_cli.core import catalog;print(len(catalog.all_entries()))"; echo EXIT=$?</code><br>
-<code># 1. exemplar counts per set</code><br>
-<code>python3 -c "</code><br>
-<code>import json</code><br>
-<code>for f in ['tests/eval/golden.jsonl','tests/eval/golden-natural.jsonl']:</code><br>
-<code>    r=[json.loads(l) for l in open(f) if l.strip() and not l.startswith('#')]</code><br>
-<code>    print(f,'rows:',len(r),'with exemplar:',sum(1 for x in r if x.get('exemplar')))"; echo EXIT=$?</code><br>
-<code># 2. the harness's own list of undecided gated rows</code><br>
-…
-
-<b>What adversarial verification corrected.</b> This card's numbers are the re-measured ones, not the ones first reported.
-
-The defect is real and the spine is exact, but five figures are stale by one corpus refresh. The finding measured on the 10,152-entry corpus CLAUDE.md documents; committed <code>tests/eval/taps.txt</code> moved on 2026-09-01 (cbc0a58b) and now materializes 10,731 entries, which is what <code>make eval</code>/CI builds today. Corrected values, all measured by me on that corpus:
-
-- hit@1 credits: 43/91 = 0.473 -&gt; <b>44/91 = 0.484</b> - un-adjudicated share of credits: 23.3% -&gt; <b>22.7%</b> (the count stays <b>10</b>) - floor headroom: 6.60 queries -&gt; <b>7.60 queries</b> - worst-case bound stripping all 10: 33/91 = 0.363 -&gt; <b>34/91 = 0.374</b> (still below the 0.400 floor) - BM25 four-metric line: 0.852 / 0.473 / 0.605 / 0.657 -&gt; <b>0.841 / 0.484 / 0.607 / 0.655</b>
-
-The finding's numbers are correct for the corpus it names and it disclosed the provenance honestly; they are nonetheless the wrong numbers to print about "the set the required eval gate floors", because the gate no longer builds that corpus. A card must publish the 10,731 figures and name the corpus.
-
-Nothing else in the finding is wrong. 0/91 vs 50/50, the 27 undecided rows / 62 candidate bodies, the same 10 names and 10 queries, the skill-creator worked example, the not-carded check, and the "realistic shift is 1-2 queries, the floor still passes" bound all hold on BOTH corpora.
-
-<b>What is NOT established.</b> Recorded because a card that overstates its own evidence is worse than no card.
-
-1. WHAT IS CORPUS-INDEPENDENT (identical on 10,152 and 10,731): exemplar counts 0/91 and 50/50; 27 undecided worksheet rows over 62 candidate bodies; the same 10 ambiguous-credit names and the same 10 queries; the skill-creator example. Only the credit count and the two ratios moved. Build the card on the invariant half and quote the 10,731 figures for the rest.
-
-2. TRAP FOR THE NEXT VERIFIER: the shared read-only <code>$TMPDIR/eval-home</code> is a PRE-REFRESH corpus — its <code>.eval-corpus-ready</code> sentinel digest does not match <code>sha256(tests/eval/taps.txt)</code>, and 10 of its 20 clones sit at a different commit. Anyone who measures only there reproduces the finding's old numbers verbatim and marks it CONFIRMED without noticing. Rebuilding at the committed pins in a disposable HOME took ~3 minutes and is what produced the correction.
-
-3. DO NOT COPY Makefile:126, which still comments "over twenty it scores 0.863 / 0.473 / 0.607 / 0.662". That matches neither corpus (10,152 gives 0.852/0.473/0.605/0.657; 10,731 gives 0.841/0.484/0.607/0.655). CLAUDE.md:67's figures are right for the old corpus and now also stale.
-
-4. SCOPE OF MY REPRO: I verified the BM25 engine only (<code>--engines bm25</code>), which is what the gate floors; I did not build a dense store (impossible here). I did not run <code>make eval</code> end to end, only its second command against a corpus I materialised with its first.
-
-5. NOT A DEFECT IN THE HARNESS: <code>exemplar_worksheet</code> (scripts/eval_retrieval.py:219) and exemplar grading work correctly and fail loudly on a bad pin.
-
-<b>Why it is worth doing.</b> golden.jsonl is the set the required <code>eval</code> gate floors, so it is the only one that can block a merge. 23.3% of its hit@1 credits are awarded on a name that maps to several genuinely different skills, and the floor's whole margin (6.60 queries) is narrower than the un-adjudicated credit count (10). That does not mean retrieval is worse than reported — the realistic shift is 1-2 queries — it means the published margin cannot be read as precision about the intended skill, so anyone tuning blend weights or pool depth against this gate inherits an unquantified slack.
-
-<em>Found by an automated audit of retrieval/eval quality, the search &amp; browse surfaces, and first-run onboarding; every finding was then re-measured from scratch by an independent adversarial verifier whose instruction was to refute it. Verdict: <b>CORRECTED</b>. No fix is prescribed here — the measurement is the contribution.</em></p>
-        <div class="meta">
-          <span class="m">Complexity <b>M</b></span>
-          <span class="m">Impact <b>Med</b></span>
-          <span class="m">Wow <b class="wow">★★★</b></span>
-          <span class="m">CLAUDE.md presents exemplar pinning as a live migration — "relevance is still decided…</span>
-        </div>
-      </article>
-
       <article class="cap rcard" id="info-deps-materialize-a-dir-they-then-reject">
         <div class="head"><span class="pill shipped">Shipped</span><span class="cat">UX · Bug</span></div>
         <h3><code>boost info</code>/<code>deps</code> on a not-installed rule or workflow widens the tap's sparse cone for a directory <code>source_dir_for</code> immediately rejects</h3>
@@ -1672,44 +1604,6 @@ Nothing else in the finding is wrong. 0/91 vs 50/50, the 27 undecided rows / 62
         </div>
       </article>
 
-      <article class="cap rcard" id="required-eval-gate-still-grades-by-name">
-        <div class="head"><span class="pill planned">Planned</span><span class="cat">Tech-debt</span></div>
-        <h3>The exemplar migration never reached the query set the required gate runs: 0 of 91 rows, and 27 of them have an ambiguous target on the gate's own corpus</h3>
-        <p><b>Measured.</b> Both invocations of the merge-blocking retrieval gate — the Makefile <code>eval</code> target and ci.yml's "retrieval quality gate" step — run <code>scripts/eval_retrieval.py</code> with no <code>--golden</code>, so both take <code>DEFAULT_GOLDEN = tests/eval/golden.jsonl</code>; 0 of that file's 91 rows carry an <code>exemplar</code>, and on the gate's own 20-tap corpus 27 of the 91 rows have at least one target name that resolves to more than one content digest (21 of 77 distinct target names are ambiguous).
-
-<b>Reproduce it.</b>
-
-<code>cd &lt;repo&gt;</code><br>
-<code>python3 -c "import json; rows=[json.loads(l) for l in open('tests/eval/golden.jsonl') if l.strip() and not l.startswith('#')]; print(len(rows),'rows;',sum(1 for r in rows if r.get('exemplar')),'with exemplar')"</code><br>
-<code>python3 -c "import json; rows=[json.loads(l) for l in open('tests/eval/golden-natural.jsonl') if l.strip() and not l.startswith('#')]; print(len(rows),'rows;',sum(1 for r in rows if r.get('exemplar')),'with exemplar')"</code><br>
-<code>grep -n 'DEFAULT_GOLDEN' scripts/eval_retrieval.py</code><br>
-<code>sed -n '134,139p' Makefile</code><br>
-<code>sed -n '269,279p' .github/workflows/ci.yml</code><br>
-<code>export BOOST_HOME=$TMPDIR/eval-home</code><br>
-<code>python3 -c "</code><br>
-<code>import json,glob,collections,os</code><br>
-<code>n2c=collections.defaultdict(set)</code><br>
-…
-
-<b>Verification found nothing to correct.</b> Every number, <code>file:line</code> and command output above was independently re-derived and matched exactly.
-
-<b>What is NOT established.</b> Recorded because a card that overstates its own evidence is worse than no card.
-
-Severity lowered from the claim's High to Medium on a measurement the claim did not make. I inspected the bodies behind the ambiguous names and they are two different things. Genuine homonyms exist — <code>brand-guidelines</code> is Anthropic brand colors vs OpenAI brand colors vs Sentry *copy writing* (three different jobs), and <code>pdf</code> is a general PDF skill vs a workflow that converts PDF to Markdown with one specific tool — but they are a minority, and neither appears among the rank-1 hits. The names that actually score hit@1 through an ambiguous target (skill-creator, slack-gif-creator, theme-factory, prompt-engineering, mcp-builder) are re-publications of one upstream skill across anthropics/skills, composio-community/awesome-codex-skills and sickn33/antigravity-awesome-skills, differing by a few bytes. A "wrong body" there is still a correct answer to the user, so the looseness is much smaller in practice than in principle. I did not adopt the shipped card's "the looseness was latent" finding as a reason: that was measured at hit@1 0.160, where there was nothing for ambiguity to inflate, whereas this set measures 0.484 — a different regime, so it does not transfer.
-
-Floor-margin exposure, with that qualification attached: hit@1 measures 44/91 = 0.484 against a floor needing 37 rows (margin 7 rows) with 10 rank-1 hits landing on an ambiguous name; recall@10 measures 77/91 = 0.846 against a floor needing 71 rows (margin 6 rows) with 20 rows matched only via ambiguous names.
-
-<b>Why it is worth doing.</b> This is the only gate in <code>make check</code> that can fail a merge on retrieval quality, and every retrieval decision validated against it (RRF fusion vs preferring dense, pool depth, whether the LLM rerank earns its keep) inherits the looseness. The project already established the fix and built the harness for it; the required set simply never got migrated, and because the card is closed as shipped nothing tracks the remaining work on the set that matters most.
-
-<em>Found by an automated audit of retrieval/eval quality, the search &amp; browse surfaces, and first-run onboarding; every finding was then re-measured from scratch by an independent adversarial verifier whose instruction was to refute it. Verdict: <b>CONFIRMED</b>. No fix is prescribed here — the measurement is the contribution.</em></p>
-        <div class="meta">
-          <span class="m">Complexity <b>M</b></span>
-          <span class="m">Impact <b>Med</b></span>
-          <span class="m">Wow <b class="wow">★★★</b></span>
-          <span class="m">CLAUDE.md states "Relevance is still decided by name (or by content class when a gold…</span>
-        </div>
-      </article>
-
       <article class="cap rcard" id="stale-tap-hint-dead-for-tap-only-installs">
         <div class="head"><span class="pill shipped">Shipped</span><span class="cat">Tech-debt</span></div>
         <h3>The "taps last refreshed N days ago" hint can never fire on a machine that tapped and never ran <code>boost update</code> — the only writer of the marker is <code>update</code> itself</h3>