boost-skill-cli 1.2.152__tar.gz → 1.2.153__tar.gz

{boost_skill_cli-1.2.152 → boost_skill_cli-1.2.153}/CLAUDE.md

@@ -122,9 +122,24 @@ collapse and homonyms do not. Relevance is still decided by name (or by content
 class when a golden row pins an `exemplar`), so the sets can migrate a row at a
 time.
 
-Each floor sits ~10% under its measured value — loose enough that upstream drift
-can't flake the build, tight enough to catch a collapse. Regression-vs-baseline
-stays relaxed (`--regression-eps 1`), so the absolute floors are the real gate.
+The floors are fixed absolute values, and the uniform "~10%" margin this file
+claimed from 170d52c0 (2026-07-31) onward was never one of them. They sit under
+their measured values (7.2% / 17.3% / 14.3% / 11.5%), recall@k the tightest —
+loose enough that upstream drift can't flake the build, tight enough to catch a
+collapse, but a 2.39x spread rather than one number, and the tightest of them
+is worth 5.5 queries of 91. The spread is the design working, not drifting: the
+monthly refresh re-measures the row and deliberately leaves the floors where
+they are, so every refresh that improves a metric widens that metric's margin.
+Floors are re-stated after a refresh, never re-derived — a floor that follows
+its own measurement ratchets down with the corpus and ends up measuring the
+corpus. `test_corpus_prose.py` holds every margin above to `baseline.json`, to
+`golden.jsonl` and to the argv of the `Makefile`'s own `eval` recipe — read
+through `eval_retrieval.build_parser()` rather than matched, because the
+tightest floor is `--fail-under` and not a `--floor`, and because a tab-indented
+comment is argv to a regex and a no-op to make. So a refresh that moves the row
+fails the build until someone writes the new margin down and looks at it.
+Regression-vs-baseline stays relaxed (`--regression-eps 1`), so the absolute
+floors are the real gate.
 
 **Every row of `taps.txt` pins a commit SHA**, and `scripts/eval_corpus.py`
 checks each clone out at it — most of the corpus is one third-party repo, so an

{boost_skill_cli-1.2.152 → boost_skill_cli-1.2.153}/Makefile

@@ -127,9 +127,14 @@ dist-check:
 # which is what tests/eval/baseline.json records at the current pins. The
 # second set is what a real user sees, so flooring against the first was
 # measuring the corpus rather than the retrieval — three of the four old floors
-# fail outright once the corpus is realistic. Each floor now sits ~10% below
-# its measured value: loose enough that a query or two cannot flake the build,
-# tight enough that a collapse fails it. Regression-vs-baseline stays relaxed
+# fail outright once the corpus is realistic. The four floors sit a margin
+# under their measured values (7.2% / 17.3% / 14.3% / 11.5%), recall@k the
+# tightest at 5.5 queries of 91: loose enough that a query or two cannot flake
+# the build, tight enough that a collapse fails it. That margin is not uniform
+# and never was -- it is a 2.39x spread, because the row moves with every
+# monthly refresh and the floors deliberately do not. So it is quoted rather
+# than asserted, and tests/unit/test_corpus_prose.py fails until the new
+# spread is written here. Regression-vs-baseline stays relaxed
 # (--regression-eps 1); the absolute floors are the real gate. (It was relaxed
 # while the corpus tracked upstream HEAD. tests/eval/taps.txt pins a commit per
 # row now, so that is no longer the reason.)

{boost_skill_cli-1.2.152 → boost_skill_cli-1.2.153}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.2.152
+Version: 1.2.153
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License-Expression: Apache-2.0

{boost_skill_cli-1.2.152 → boost_skill_cli-1.2.153}/boost_cli/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '1.2.152'
-__version_tuple__ = version_tuple = (1, 2, 152)
+__version__ = version = '1.2.153'
+__version_tuple__ = version_tuple = (1, 2, 153)
 
 __commit_id__ = commit_id = None

{boost_skill_cli-1.2.152 → boost_skill_cli-1.2.153}/boost_skill_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.2.152
+Version: 1.2.153
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License-Expression: Apache-2.0

{boost_skill_cli-1.2.152 → boost_skill_cli-1.2.153}/docs/roadmap/items/eval-floor-calibration-stale-after-pin-refresh.md

@@ -1,16 +1,16 @@
 ---
 id: eval-floor-calibration-stale-after-pin-refresh
 board: code
-section: planned
-status: planned
+section: shipped
+status: shipped
 category: Quality · Retrieval eval
 complexity: M
 impact: Med
 wow: 3
 note: CLAUDE.md states the gate's four floors sit "~10% under" measured values of 0.852 / 0…
 order: 210
-owner:
-pr:
+owner: loop/eval-floor-calibration
+pr: "993"
 title: The 2026-09-01 pin refresh moved the corpus and re-baselined it, but nothing re-derives the floors — CLAUDE.md's "~10% under measured" is now 7.2%–17.3%
 ---
 <b>Measured.</b> The required gate's recall floor has 5.52 queries of headroom out of 91 ((0.8407 - 0.78) x 91), against the 6.55 that CLAUDE.md's published 0.852 implies — one golden query of margin the documentation says exists and does not, because commit cbc0a58b re-baselined the corpus to 10,731 entries and left CLAUDE.md, taps.txt and eval_corpus.py all stating 10,152 / 0.852.
@@ -32,7 +32,7 @@ title: The 2026-09-01 pin refresh moved the corpus and re-baselined it, but noth
 
 <b>What adversarial verification corrected.</b> This card's numbers are the re-measured ones, not the ones first reported.
 
-1. CAUSAL FRAMING IS WRONG (the headline correction). The title and claim say the 2026-09-01 pin refresh is what made "~10% under" untrue. It was untrue the day it was written. <code>git log -S'~10% under'</code> traces the sentence to 170d52c0 (2026-07-31), whose own commit message says the floors were "re-derived against the 20-tap numbers at the same ~10% relative headroom" against measured 0.863 / 0.473 / 0.607 / 0.662 — at which point the gaps were 9.6% / 15.4% / 14.3% / 12.4%, a 1.60x spread. The refresh WIDENED a spread (1.60x -&gt; 1.82x -&gt; 2.39x); it did not create one. Correct statement: "the floors were never uniformly ~10% under, and the refresh widened the spread from 1.82x to 2.39x."
+1. CAUSAL FRAMING IS WRONG (the headline correction). The title and claim say the 2026-09-01 pin refresh is what made "~10% under" untrue. It was untrue the day it was written. <code>git log -S'~10% under'</code> traces the sentence to 170d52c0 (2026-07-31), whose own commit message says the floors were "re-derived against the 20-tap numbers at the same ~10% relative headroom" against measured 0.863 / 0.473 / 0.607 / 0.662 — at which point the gaps were 9.6% / 15.3% / 14.4% / 12.4%, a 1.60x spread. (This line first read 9.6% / 15.4% / 14.3% / 12.4%; recomputing from 170d52c0's own <code>baseline.json</code> and <code>Makefile</code> gives 9.5796 / 15.3488 / 14.3831 / 12.4408, so two digits were transposed.) The spread widened twice and by two different mechanisms, only one of which is a refresh: 1.60x at 170d52c0, 1.82x at d8da2e71 — <em>de-duplicate the ranked list by body, not by name</em>, a ranker correctness fix (#411) — and 2.39x at the 2026-09-01 refresh cbc0a58b. Correct statement: "the floors were never uniformly ~10% under, and nothing since has narrowed the spread."
 
 2. "not the 6.60 the documented number implies" — 6.60 matches nothing. CLAUDE.md's published 0.852 gives (0.852-0.78) x 91 = 6.55 queries; the unrounded pre-refresh baseline 0.8406... gives 6.52. Use 6.55. (The optimism is therefore ~1.03 queries, not 1.1.)
 
@@ -53,3 +53,9 @@ SCOPE LIMIT #2: the shared $TMPDIR/eval-home is materialised at the PRE-refresh
 <b>Why it is worth doing.</b> The "~10% under" statement is the entire published justification for where the four floors sit — CLAUDE.md calls it "loose enough that upstream drift can't flake the build, tight enough to catch a collapse." That is a claim about a margin, and the margin now differs 2.4x across the four metrics with recall the tightest at 5.52 queries out of 91. Because the monthly refresh re-baselines but never re-derives or re-states the floors, and prompts a human only when the refreshed corpus FAILS, every future passing refresh decays the calibration further with no signal at all.
 
 <em>Found by an automated audit of retrieval/eval quality, the search &amp; browse surfaces, and first-run onboarding; every finding was then re-measured from scratch by an independent adversarial verifier whose instruction was to refute it. Verdict: <b>CORRECTED</b>. No fix is prescribed here — the measurement is the contribution.</em>
+
+<b>What the fix turned out to be, and what the card got wrong about scope.</b> Half of this card was already fixed when it was picked up. The figure half — "CLAUDE.md, taps.txt and eval_corpus.py all stating 10,152 / 0.852" — is stale: <code>tests/unit/test_corpus_prose.py</code> now holds every one of those figures to <code>taps.txt</code>'s own rows and to <code>baseline.json</code>, so CLAUDE.md reads 10,731 and 0.841 / 0.484 / 0.607 / 0.655 and cannot drift again. The card's line references (CLAUDE.md:67,84) no longer point at anything. What survived is the card's own stated core, and it is the harder half: the <em>characterisation</em> "~10% under" was prose, was false the day it was written, and no mechanism touched it.
+
+<b>The remedy is the card's own, not the one the finding's wording invites.</b> No floor moves and no headroom gate is added — a floor that follows its own measurement ends up measuring the corpus, which is exactly what 170d52c0 warned about. Instead the true margin is written down at measured precision (7.2% / 17.3% / 14.3% / 11.5%, a 2.39x spread, recall@k binding at 5.5 queries of 91) and held by the <code>QUOTED</code> mechanism that already holds the figures beside it. A refresh that moves the row now fails the build until a human restates the margin and looks at it — which is the signal the card says is missing, without touching the gate it says not to touch.
+
+<b>Why the required set drifted and the advisory set did not.</b> The contrast is the tell. <code>make eval-natural</code>'s floors really are ~10% under (11.1 / 12.5 / 11.4 / 11.2, a 1.12x spread) because <code>tests/unit/test_eval_corpus.py</code> moves them with the row in both directions. The required set, where the claim actually matters, had no such mechanism — so the one place the margin was load-bearing is the one place nothing checked it.

{boost_skill_cli-1.2.152 → boost_skill_cli-1.2.153}/docs/roadmap/items/eval-gate-scores-a-body-less-corpus.md

@@ -52,6 +52,6 @@ WHY Med, NOT High — two facts I verified that cap the blast radius: 1. CI cann
 
 SCOPE LIMITS OF MY REPRO — a card author must not blur these: - I measured the FULLY-missing case. The "half-materialised tap tree" phrasing in <code>why_it_matters</code> was NOT measured by me; degradation is presumably proportional but is unverified.
 
-<b>Why it is worth doing.</b> The gate whose whole job is to say "retrieval still works over a real corpus" returns a confident, better-than-baseline PASS over a corpus that is name+description only — 95% of its text absent. A developer who reclaims the 265 MB <code>repos/</code> tree, or whose tap tree is half-materialised, gets four green floors that attest to nothing, and the greener number makes the loss look like an improvement rather than a defect. Every downstream floor calibration ("each floor sits ~10% under its measured value") is stated against 0.473/0.605/0.657, which the body-less run clears by 48%/33%/25% instead.
+<b>Why it is worth doing.</b> The gate whose whole job is to say "retrieval still works over a real corpus" returns a confident, better-than-baseline PASS over a corpus that is name+description only — 95% of its text absent. A developer who reclaims the 265 MB <code>repos/</code> tree, or whose tap tree is half-materialised, gets four green floors that attest to nothing, and the greener number makes the loss look like an improvement rather than a defect. Every downstream floor calibration is stated against 0.473/0.605/0.657, which the body-less run clears by 48%/33%/25% instead. (This line quoted CLAUDE.md's "each floor sits ~10% under its measured value" as the live convention. That sentence was retired in PR #993: the margins were never uniform — 7.2% / 17.3% / 14.3% / 11.5% at the current pins — and CLAUDE.md now quotes all four and holds them.)
 
 <em>Found by an automated audit of retrieval/eval quality, the search &amp; browse surfaces, and first-run onboarding; every finding was then re-measured from scratch by an independent adversarial verifier whose instruction was to refute it. Verdict: <b>CORRECTED</b>. No fix is prescribed here — the measurement is the contribution.</em>

{boost_skill_cli-1.2.152 → boost_skill_cli-1.2.153}/docs/roadmap.html

@@ -140,9 +140,9 @@
   <section class="block reveal">
     <div class="stats">
       <!-- ROADMAP:stats -->
-      <div class="stat"><b>24</b><span>Shipped</span></div>
+      <div class="stat"><b>25</b><span>Shipped</span></div>
       <div class="stat"><b>1</b><span>Next up</span></div>
-      <div class="stat"><b>77</b><span>Planned</span></div>
+      <div class="stat"><b>76</b><span>Planned</span></div>
       <div class="stat"><b>505</b><span>Loop finds</span></div>
       <!-- /ROADMAP:stats -->
     </div>
@@ -420,6 +420,18 @@
         </div>
       </article>
 
+      <article class="cap rcard" id="eval-floor-calibration-stale-after-pin-refresh">
+        <div class="head"><span class="pill shipped">Shipped</span><span class="cat">Quality · Retrieval eval</span></div>
+        <h3>The 2026-09-01 pin refresh moved the corpus and re-baselined it, but nothing re-derives the floors — CLAUDE.md's "~10% under measured" is now 7.2%–17.3%</h3>
+        <p class="cardbody"><a class="writeup" href="https://github.com/jonnyeclectic/boost/blob/main/docs/roadmap/items/eval-floor-calibration-stale-after-pin-refresh.md">Write-up &middot; eval-floor-calibration-stale-after-pin-refresh.md</a></p>
+        <div class="meta">
+          <span class="m">Complexity <b>M</b></span>
+          <span class="m">Impact <b>Med</b></span>
+          <span class="m">Wow <b class="wow">★★★</b></span>
+          <span class="m">CLAUDE.md states the gate's four floors sit "~10% under" measured values of 0.852 / 0…</span>
+        </div>
+      </article>
+
       <article class="cap rcard" id="install-cannot-tell-a-skill-needs-another-one">
         <div class="head"><span class="pill shipped">Shipped</span><span class="cat">Install · Research</span></div>
         <h3>Research: how <code>boost install</code> could tell that a skill needs something else installed first</h3>
@@ -1208,57 +1220,6 @@ NOT ALREADY CARDED, but read the existing card first.
         </div>
       </article>
 
-      <article class="cap rcard" id="eval-floor-calibration-stale-after-pin-refresh">
-        <div class="head"><span class="pill planned">Planned</span><span class="cat">Quality · Retrieval eval</span></div>
-        <h3>The 2026-09-01 pin refresh moved the corpus and re-baselined it, but nothing re-derives the floors — CLAUDE.md's "~10% under measured" is now 7.2%–17.3%</h3>
-        <p><b>Measured.</b> The required gate's recall floor has 5.52 queries of headroom out of 91 ((0.8407 - 0.78) x 91), against the 6.55 that CLAUDE.md's published 0.852 implies — one golden query of margin the documentation says exists and does not, because commit cbc0a58b re-baselined the corpus to 10,731 entries and left CLAUDE.md, taps.txt and eval_corpus.py all stating 10,152 / 0.852.
-
-<b>Reproduce it.</b>
-
-<code>cd &lt;repo&gt;</code><br>
-<code># 1. the refresh commit moved pins + baseline and nothing else</code><br>
-<code>git show cbc0a58b --stat --format='%h %ad %s' --date=short</code><br>
-<code>git show cbc0a58b -- tests/eval/baseline.json | grep -E '^[-+].*(recall|hit@1|MRR|nDCG)'</code><br>
-<code># 2. what taps.txt records NOW vs what the docs claim</code><br>
-<code>awk '!/^#/ &amp;&amp; NF&gt;=3 {s+=$3; n++} END {print "rows:", n, " recorded total entries:", s}' tests/eval/taps.txt</code><br>
-<code>grep -n '10,152\|0\.852' CLAUDE.md tests/eval/taps.txt scripts/eval_corpus.py</code><br>
-<code># 3. recompute the floor gaps against the committed post-refresh baseline</code><br>
-<code>python3 -c "</code><br>
-<code>import json</code><br>
-<code>b=json.load(open('tests/eval/baseline.json'))['sets']['golden.jsonl@a0617183f8c9']['engines']['BM25 full-content']</code><br>
-…
-
-<b>What adversarial verification corrected.</b> This card's numbers are the re-measured ones, not the ones first reported.
-
-1. CAUSAL FRAMING IS WRONG (the headline correction). The title and claim say the 2026-09-01 pin refresh is what made "~10% under" untrue. It was untrue the day it was written. <code>git log -S'~10% under'</code> traces the sentence to 170d52c0 (2026-07-31), whose own commit message says the floors were "re-derived against the 20-tap numbers at the same ~10% relative headroom" against measured 0.863 / 0.473 / 0.607 / 0.662 — at which point the gaps were 9.6% / 15.4% / 14.3% / 12.4%, a 1.60x spread. The refresh WIDENED a spread (1.60x -&gt; 1.82x -&gt; 2.39x); it did not create one. Correct statement: "the floors were never uniformly ~10% under, and the refresh widened the spread from 1.82x to 2.39x."
-
-2. "not the 6.60 the documented number implies" — 6.60 matches nothing. CLAUDE.md's published 0.852 gives (0.852-0.78) x 91 = 6.55 queries; the unrounded pre-refresh baseline 0.8406... gives 6.52. Use 6.55. (The optimism is therefore ~1.03 queries, not 1.1.)
-
-3. "2.4x spread" -&gt; 2.39x (rounding, harmless, but state it as 2.39x on a card).
-
-4. DIRECTION OF DRIFT IS NOT UNIFORM. "every future passing refresh decays the calibration further" is true only of the SPREAD. This refresh moved two floors tighter (recall 8.4%-&gt;7.2%, nDCG 11.8%-&gt;11.5%) and two LOOSER (hit@1 15.3%-&gt;17.3%, MRR 14.0%-&gt;14.3%). A card must not say "the floors tightened."
-
-5.
-
-<b>What is NOT established.</b> Recorded because a card that overstates its own evidence is worse than no card.
-
-SCOPE THE REMEDY CAREFULLY — the finding's phrase "nothing re-derives the floors" invites exactly the fix 170d52c0 warns against: auto-moving floors on every refresh ("Lowering a threshold deserves suspicion... a floor calibrated on an unrepresentative corpus measures the corpus"). The repo's design is deliberately fixed absolute floors with a relaxed regression-vs-baseline. The CONFIRMED defect is narrower and should be carded as such: the measured values and the "~10%" characterisation are stated in seven places across four files (CLAUDE.md:67,84; tests/eval/taps.txt:25,43; scripts/eval_corpus.py:30,108 and the runtime-printed operator message at :331-334), nothing re-states or guards them after a PASSING refresh, and no test asserts any of them.
-
-SCOPE LIMIT OF MY REPRO #1: I did not re-measure 0.8407 / 0.4835 / 0.6065 / 0.6552. Those are the committed baseline written by cbc0a58b — which is also the finder's only source. Verifying them empirically needs a re-tap at the new pins, which is forbidden here (shared read-only corpus) and needs network. Every derived percentage in this verification inherits that.
-
-SCOPE LIMIT #2: the shared $TMPDIR/eval-home is materialised at the PRE-refresh pins — live total 10,152 vs taps.txt's recorded 10,731, with the same 10 taps PIN-DIFFERS. <code>make eval</code> against it would trip eval_corpus.py's CORPUS DRIFT check, not run the gate.
-
-<b>Why it is worth doing.</b> The "~10% under" statement is the entire published justification for where the four floors sit — CLAUDE.md calls it "loose enough that upstream drift can't flake the build, tight enough to catch a collapse." That is a claim about a margin, and the margin now differs 2.4x across the four metrics with recall the tightest at 5.52 queries out of 91. Because the monthly refresh re-baselines but never re-derives or re-states the floors, and prompts a human only when the refreshed corpus FAILS, every future passing refresh decays the calibration further with no signal at all.
-
-<em>Found by an automated audit of retrieval/eval quality, the search &amp; browse surfaces, and first-run onboarding; every finding was then re-measured from scratch by an independent adversarial verifier whose instruction was to refute it. Verdict: <b>CORRECTED</b>. No fix is prescribed here — the measurement is the contribution.</em></p>
-        <div class="meta">
-          <span class="m">Complexity <b>M</b></span>
-          <span class="m">Impact <b>Med</b></span>
-          <span class="m">Wow <b class="wow">★★★</b></span>
-          <span class="m">CLAUDE.md states the gate's four floors sit "~10% under" measured values of 0.852 / 0…</span>
-        </div>
-      </article>
-
       <article class="cap rcard" id="eval-gate-scores-a-body-less-corpus">
         <div class="head"><span class="pill shipped">Shipped</span><span class="cat">Quality · Retrieval eval</span></div>
         <h3><code>make eval</code> scores a corpus with every SKILL.md body missing, reports all four floors PASS, and scores HIGHER than the real corpus</h3>

{boost_skill_cli-1.2.152 → boost_skill_cli-1.2.153}/tests/unit/test_corpus_prose.py

@@ -31,6 +31,7 @@ import pytest
 _ROOT = Path(__file__).resolve().parents[2]
 _TAPS = _ROOT / "tests" / "eval" / "taps.txt"
 _BASELINE = _ROOT / "tests" / "eval" / "baseline.json"
+_GOLDEN = _ROOT / "tests" / "eval" / "golden.jsonl"
 
 pytestmark = pytest.mark.skipif(
     not (_TAPS.exists() and _BASELINE.exists()),
@@ -48,6 +49,12 @@ QUOTED = (
     ("CLAUDE.md", "over the six alone ({targets} entries"),
     ("CLAUDE.md", "**{bm25}**"),
     ("CLAUDE.md", "recorded BM25 row ({natural_bm25})"),
+    ("CLAUDE.md", "under their measured values ({headroom}), {binding} the tightest"),
+    ("CLAUDE.md", "a {spread} spread rather than one number"),
+    ("CLAUDE.md", "the tightest of them is worth {slack}"),
+    ("Makefile", "under their measured values ({headroom}), {binding} the"),
+    ("Makefile", "tightest at {slack}:"),
+    ("Makefile", "it is a {spread} spread"),
     ("Makefile", "Over the six ({targets} entries)"),
     ("Makefile", "over twenty it scores {bm25}"),
     ("Makefile", "records ({natural_bm25} at the current pins"),
@@ -122,6 +129,9 @@ def _figures() -> dict[str, str]:
     # The rows above the scale divider are the ones holding golden targets.
     head = text.split("\n# --- scale", 1)[0]
     bm25 = _bm25()
+    floors = _floors()
+    gaps = _headroom(bm25, floors)
+    tightest = _METRICS[gaps.index(min(gaps))]
     return {
         "total": f"{sum(_counts(text)):,}",
         "targets": f"{sum(_counts(head)):,}",
@@ -130,6 +140,32 @@ def _figures() -> dict[str, str]:
         # `make eval-natural`'s floors are calibrated on the figures quoted.
         "natural_bm25": _four(_bm25("golden-natural.jsonl")),
         "recall": "%.3f" % bm25["recall@k"],
+        # The margin each floor actually has, rather than the single "~10%"
+        # the prose asserted from 170d52c0 (2026-07-31) onward. It was never
+        # uniform: at that twenty-tap calibration the gaps were 9.6 / 15.3 /
+        # 14.4 / 12.4 (recomputed from 170d52c0's own baseline and Makefile —
+        # the roadmap card transposes two of those digits),
+        # and the monthly refresh widens the spread because it re-baselines the
+        # row and leaves the floors where they are, by design. Quoting the four
+        # separately is what makes that visible; holding them here is what
+        # makes a refresh restate them.
+        "headroom": " / ".join("%.1f%%" % g for g in gaps),
+        # Which floor is closest to the row, and so the one a corpus move
+        # actually trips. A judgement stated beside held figures has to be held
+        # too, or it is the next sentence to go quietly stale.
+        "binding": tightest,
+        # How wide the spread is. The claim being replaced was that there was
+        # no spread, so the number that refutes it has to be held like the
+        # rest -- an unheld "2.4x" beside a held "7.2% / 17.3%" is the same
+        # defect one sentence along. Two decimals, not one: the verification
+        # behind this card rejected "2.4x" for "2.39x", and a figure quoted
+        # to the precision it was measured at cannot be argued with.
+        "spread": "%.2fx" % (max(gaps) / min(gaps)),
+        # The binding floor's margin in queries rather than percent. A
+        # percentage does not say whether a margin is one flaky query or ten;
+        # over this set, 1 query is 1.1 points of recall@k.
+        "slack": "%.1f queries of %d" % (
+            (bm25[tightest] - floors[tightest]) * _queries(), _queries()),
     }
 
 
@@ -148,6 +184,99 @@ def _max_share() -> float:
     return float(mod.MAX_ROW_SHARE)
 
 
+#: The Makefile recipe whose `--floor` arguments the required gate runs under.
+#: Only this one is parsed: `tests/unit/test_eval_corpus.py` already holds the
+#: Makefile, ci.yml and eval-corpus-refresh.yml to one argv, so a floor read
+#: here is the floor all three use.
+_EVAL_RECIPE = "eval"
+#: The four metrics, in the order the prose quotes them.
+_METRICS = ("recall@k", "hit@1", "MRR", "nDCG@k")
+
+
+def _argv_reader():
+    """`tests/unit/test_eval_corpus.py`, loaded for its argv reader.
+
+    The floors have to be read the way the gate reads them, and the reader
+    already exists one file over: `_recipe` slices the Makefile to one
+    target, `_invocation` joins line continuations, drops comment lines and
+    shlex-splits, and `_meaning` hands the words to
+    `eval_retrieval.build_parser()` itself.
+
+    Re-implementing that here with a regex put a hole in this very check
+    twice. First an unindented comment block, which sits *above* the next
+    target, so slicing to `eval-natural:` swallowed all of its prose. Then —
+    after that was fixed by taking tab-indented lines only — a **tab-indented
+    comment inside the recipe**, which make ignores and a regex does not:
+
+        eval:
+                $(PY) scripts/eval_retrieval.py ... --floor hit@1=0.30 \\
+                # relaxed from --floor hit@1=0.40
+
+    make runs the 0.30 gate; a last-match-wins regex reports 0.40, so the
+    published margin is held to a floor nothing enforces and nothing is red.
+    A third spelling was waiting behind both: argparse accepts
+    `--fail-under=0.60`, and a regex keyed on whitespace never sees it.
+    Delegating cures all three at once, and any fourth, because the thing
+    doing the parsing is the parser.
+
+    Loaded by path rather than imported by name, so this does not depend on
+    `tests/unit` being on `sys.path`, and under a private module name so it
+    cannot collide with pytest's own import of the same file.
+    """
+    spec = importlib.util.spec_from_file_location(
+        "_corpus_prose_argv", _ROOT / "tests" / "unit" / "test_eval_corpus.py")
+    assert spec and spec.loader
+    mod = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(mod)
+    return mod
+
+
+def _floors_in(body: str) -> dict[str, float]:
+    """The floors one recipe body passes, as the gate's own parser sees them.
+
+    recall@k is `--fail-under` rather than `--floor recall@k=`, which is why
+    it is lifted out separately here and why the prose below says "argv"
+    rather than naming one flag.
+    """
+    reader = _argv_reader()
+    meaning = reader._meaning(reader._invocation(body))
+    floors = {"recall@k": float(meaning["fail_under"])}
+    floors.update({k: float(v) for k, v in meaning["floor"].items()})
+    return floors
+
+
+def _floors() -> dict[str, float]:
+    """The required gate's four floors, read from the Makefile's own argv."""
+    reader = _argv_reader()
+    marker = "\n%s:" % _EVAL_RECIPE
+    count = (_ROOT / "Makefile").read_text(encoding="utf-8").count(marker)
+    assert count == 1, (
+        "`%s` is defined %d times; make runs the LAST recipe for a duplicated "
+        "target and this reads the first, so the margin below would be held "
+        "to floors the gate does not run" % (_EVAL_RECIPE, count))
+    return _floors_in(reader._recipe(_EVAL_RECIPE))
+
+
+def _headroom(row: dict[str, float], floors: dict[str, float]) -> list[float]:
+    """How far under the measured row each floor sits, as a percentage."""
+    return [(row[m] - floors[m]) / row[m] * 100 for m in _METRICS]
+
+
+def _queries() -> int:
+    """How many cases the gate actually scores.
+
+    `golden.jsonl` is 102 lines and eleven of them are not queries — ten
+    comment lines and one blank — so a line count is not the query count.
+    `eval_retrieval.load_golden` skips blanks and `#` after stripping, and
+    this must skip exactly what it skips. The blank is the case a naive
+    `startswith('#')` would miss. The prose quotes
+    this figure to turn a percentage into queries, which is the only form in
+    which "is that margin big enough" is answerable.
+    """
+    return sum(1 for line in _GOLDEN.read_text(encoding="utf-8").splitlines()
+               if line.strip() and not line.strip().startswith("#"))
+
+
 def _folded(text: str) -> str:
     return re.sub(r"[ \t]*\n[ \t]*(?:#[ \t]*)?", " ", text)
 
@@ -258,6 +387,92 @@ class TestTheCheckItself:
         assert _folded("# measures\n# 10,731 entries") == \
             "# measures 10,731 entries"
 
+    def test_the_required_recipe_is_read_and_not_the_advisory_one(self):
+        # Two recipes carry `--fail-under` and `--floor hit@1=`: `eval`, which
+        # is the required gate, and `eval-natural`, which is advisory and
+        # whose floors DO follow the row. A bare search over the file returns
+        # whichever comes first, which is the right answer today by ordering
+        # alone. Slicing to the named recipe is what makes it the right answer
+        # tomorrow, so the slicing is what is pinned -- against the real
+        # Makefile, because that is the file the margin is held to.
+        reader = _argv_reader()
+        assert _floors() == _floors_in(reader._recipe("eval"))
+        assert _floors() != _floors_in(reader._recipe("eval-natural"))
+
+    def test_a_target_is_defined_once(self):
+        # make runs the LAST recipe for a duplicated target (it warns, then
+        # overrides), and every reader here takes the first. `_floors`
+        # refuses rather than reading a recipe make would not run; this pins
+        # that the real Makefile satisfies it, so the refusal stays a
+        # tripwire rather than a permanent failure.
+        text = (_ROOT / "Makefile").read_text(encoding="utf-8")
+        assert text.count("\n%s:" % _EVAL_RECIPE) == 1
+
+    def test_a_tab_indented_comment_is_not_argv(self):
+        # The second hole, found after the first was fixed. A comment line
+        # that starts with a TAB is inside the recipe and make ignores it --
+        # so a regex over the recipe text reads floors the gate does not run.
+        # Last-match-wins made it worse than a miss: the stale number won.
+        # Nothing looks broken either way, which is the whole problem.
+        body = ("\t$(PY) scripts/eval_retrieval.py --fail-under 0.78 "
+                "--floor hit@1=0.40\n"
+                "\t# relaxed from --fail-under 0.90 --floor hit@1=0.99\n")
+        assert _floors_in(body) == {"recall@k": 0.78, "hit@1": 0.40}
+
+    def test_an_equals_form_flag_is_read(self):
+        # The third spelling, and the one a regex keyed on whitespace cannot
+        # see at all: argparse accepts `--flag=value`. A regex reported the
+        # OLD floor here -- a silent fail-open in the opposite direction from
+        # the comment, and the reason this reads argv through the gate's own
+        # parser instead of matching text.
+        body = ("\t$(PY) scripts/eval_retrieval.py --fail-under=0.60 "
+                "--floor=hit@1=0.20\n")
+        assert _floors_in(body) == {"recall@k": 0.60, "hit@1": 0.20}
+
+    def test_a_line_continuation_is_joined(self):
+        # The real recipe wraps the call over three lines, so a reader that
+        # did not join continuations would see one flag and miss the rest --
+        # and `_floors` would return fewer than four metrics, which
+        # `test_a_floor_that_is_not_read_fails_the_check` catches. Pinned
+        # here so the failure names the cause.
+        body = ("\t$(PY) scripts/eval_retrieval.py --fail-under 0.78 \\\n"
+                "\t  --floor hit@1=0.40 \\\n"
+                "\t  --floor MRR=0.52\n")
+        assert _floors_in(body) == {
+            "recall@k": 0.78, "hit@1": 0.40, "MRR": 0.52}
+
+    def test_a_repeated_flag_is_read_the_way_argparse_reads_it(self):
+        # argparse takes the LAST occurrence of a repeated flag. Reading
+        # `--fail-under` with re.search took the first, so a duplicate would
+        # have held the prose to a number the gate does not use.
+        body = ("\t$(PY) scripts/eval_retrieval.py --fail-under 0.78 "
+                "--fail-under 0.90 --floor MRR=0.52 --floor MRR=0.61\n")
+        assert _floors_in(body) == {"recall@k": 0.90, "MRR": 0.61}
+
+    def test_a_flag_the_gate_does_not_accept_fails_here(self):
+        # Reading argv through `eval_retrieval.build_parser()` means a flag
+        # the script would reject fails in this test rather than in CI. A
+        # regex would have shrugged and returned the floors it did match.
+        with pytest.raises(SystemExit):
+            _floors_in("\t$(PY) scripts/eval_retrieval.py --no-such-flag 1\n")
+
+    def test_a_floor_that_is_not_read_fails_the_check(self):
+        # `_floors` divides by figures it parsed out of argv, so a regex that
+        # silently stops matching would not raise -- it would drop a metric
+        # and quietly narrow what the prose is held to. Four floors are what
+        # the required gate passes; fewer means the parse broke.
+        floors = _floors()
+        assert sorted(floors) == sorted(_METRICS)
+        assert all(0 < v < 1 for v in floors.values())
+
+    def test_the_query_count_skips_the_header_comment(self):
+        # golden.jsonl carries ten comment lines and a blank, so a line
+        # count is eleven queries too many. `_queries` must skip exactly what
+        # `eval_retrieval.load_golden` skips, blank included.
+        lines = _GOLDEN.read_text(encoding="utf-8").splitlines()
+        assert _queries() == sum(1 for ln in lines if ln.startswith("{"))
+        assert _queries() < len(lines)
+
 
 class TestTheShippedProse:
     @pytest.mark.parametrize("path,template", QUOTED,