boost-skill-cli 1.2.151__tar.gz → 1.2.153__tar.gz

{boost_skill_cli-1.2.151 → boost_skill_cli-1.2.153}/CLAUDE.md

@@ -122,9 +122,24 @@ collapse and homonyms do not. Relevance is still decided by name (or by content
 class when a golden row pins an `exemplar`), so the sets can migrate a row at a
 time.
 
-Each floor sits ~10% under its measured value — loose enough that upstream drift
-can't flake the build, tight enough to catch a collapse. Regression-vs-baseline
-stays relaxed (`--regression-eps 1`), so the absolute floors are the real gate.
+The floors are fixed absolute values, and the uniform "~10%" margin this file
+claimed from 170d52c0 (2026-07-31) onward was never one of them. They sit under
+their measured values (7.2% / 17.3% / 14.3% / 11.5%), recall@k the tightest —
+loose enough that upstream drift can't flake the build, tight enough to catch a
+collapse, but a 2.39x spread rather than one number, and the tightest of them
+is worth 5.5 queries of 91. The spread is the design working, not drifting: the
+monthly refresh re-measures the row and deliberately leaves the floors where
+they are, so every refresh that improves a metric widens that metric's margin.
+Floors are re-stated after a refresh, never re-derived — a floor that follows
+its own measurement ratchets down with the corpus and ends up measuring the
+corpus. `test_corpus_prose.py` holds every margin above to `baseline.json`, to
+`golden.jsonl` and to the argv of the `Makefile`'s own `eval` recipe — read
+through `eval_retrieval.build_parser()` rather than matched, because the
+tightest floor is `--fail-under` and not a `--floor`, and because a tab-indented
+comment is argv to a regex and a no-op to make. So a refresh that moves the row
+fails the build until someone writes the new margin down and looks at it.
+Regression-vs-baseline stays relaxed (`--regression-eps 1`), so the absolute
+floors are the real gate.
 
 **Every row of `taps.txt` pins a commit SHA**, and `scripts/eval_corpus.py`
 checks each clone out at it — most of the corpus is one third-party repo, so an

{boost_skill_cli-1.2.151 → boost_skill_cli-1.2.153}/Makefile

@@ -127,9 +127,14 @@ dist-check:
 # which is what tests/eval/baseline.json records at the current pins. The
 # second set is what a real user sees, so flooring against the first was
 # measuring the corpus rather than the retrieval — three of the four old floors
-# fail outright once the corpus is realistic. Each floor now sits ~10% below
-# its measured value: loose enough that a query or two cannot flake the build,
-# tight enough that a collapse fails it. Regression-vs-baseline stays relaxed
+# fail outright once the corpus is realistic. The four floors sit a margin
+# under their measured values (7.2% / 17.3% / 14.3% / 11.5%), recall@k the
+# tightest at 5.5 queries of 91: loose enough that a query or two cannot flake
+# the build, tight enough that a collapse fails it. That margin is not uniform
+# and never was -- it is a 2.39x spread, because the row moves with every
+# monthly refresh and the floors deliberately do not. So it is quoted rather
+# than asserted, and tests/unit/test_corpus_prose.py fails until the new
+# spread is written here. Regression-vs-baseline stays relaxed
 # (--regression-eps 1); the absolute floors are the real gate. (It was relaxed
 # while the corpus tracked upstream HEAD. tests/eval/taps.txt pins a commit per
 # row now, so that is no longer the reason.)

{boost_skill_cli-1.2.151 → boost_skill_cli-1.2.153}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.2.151
+Version: 1.2.153
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License-Expression: Apache-2.0

{boost_skill_cli-1.2.151 → boost_skill_cli-1.2.153}/boost_cli/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '1.2.151'
-__version_tuple__ = version_tuple = (1, 2, 151)
+__version__ = version = '1.2.153'
+__version_tuple__ = version_tuple = (1, 2, 153)
 
 __commit_id__ = commit_id = None

{boost_skill_cli-1.2.151 → boost_skill_cli-1.2.153}/boost_cli/core/catalog.py

@@ -26,6 +26,7 @@ import operator
 import os
 import re
 import sys
+from collections.abc import Iterable
 from pathlib import Path
 
 from ..errors import BoostError
@@ -115,6 +116,49 @@ def _content_digest(name: str, description: str, body: str) -> str:
     return hashlib.sha256(text.strip().encode("utf-8", "replace")).hexdigest()[:16]
 
 
+def content_unique(entries: Iterable[dict]) -> list[dict]:
+    """The first entry of each distinct ``content`` digest, in scan order.
+
+    An entry carrying no digest is always kept. Two absences are not a match:
+    a cache written before ``CACHE_FORMAT`` stamped the field, or an entry
+    synthesised without one, says nothing about what that item *is*, and
+    collapsing unknowns into each other silently deletes real items.
+
+    The digest is :func:`_content_digest` — ``name``, ``description`` and the
+    frontmatter-stripped body — which is the identity
+    :func:`rag.dedupe_by_content` collapses a ranked list on. It is not
+    ``scripts/measure_registry.py``'s digest, which hashes the body alone with
+    agent dotdir tokens normalised so one skill rendered into ``.claude/``,
+    ``.cursor/`` and ``.gemini/`` counts once. That is the right question for
+    ``est_items`` and the wrong one here: the gate scores those renders as
+    separate competing rows, so a measure that merges them cannot describe the
+    corpus it ranks.
+    """
+    seen: set[str] = set()
+    out: list[dict] = []
+    for entry in entries:
+        digest = entry.get("content")
+        if not digest:
+            out.append(entry)
+            continue
+        if digest in seen:
+            continue
+        seen.add(digest)
+        out.append(entry)
+    return out
+
+
+def distinct_content(entries: Iterable[dict]) -> int:
+    """How many distinct content identities ``entries`` holds.
+
+    ``len(content_unique(entries))``, and deliberately not
+    ``len({e["content"] for e in entries})``: the set comprehension collapses
+    every entry with no recorded digest into one ``None``, which is the one
+    answer :func:`content_unique` exists to refuse.
+    """
+    return len(content_unique(entries))
+
+
 def _entry_category(meta: dict, tap_category: str) -> str:
     """An item's own category, falling back to its tap's curated category.
 
@@ -406,9 +450,8 @@ def lint_targets(entries: list[dict], tap_root: Path,
                 "no such name%s in this tap: %s"
                 % ("" if len(unknown) == 1 else "s", ", ".join(unknown)),
                 hint="run without NAMEs to lint everything in the tap")
-    skills: list[dict] = []
+    kept: list[dict] = []
     skipped: list[dict] = []
-    seen_content: set = set()
     for entry in entries:
         if wanted and entry["name"] not in wanted:
             continue
@@ -416,12 +459,10 @@ def lint_targets(entries: list[dict], tap_root: Path,
         if kind != KIND_SKILL:
             skipped.append({"name": entry["name"], "kind": kind})
             continue
-        digest = entry.get("content")
-        if digest:
-            if digest in seen_content:
-                continue
-            seen_content.add(digest)
-        skills.append(entry)
+        kept.append(entry)
+    # One rule, one implementation: mirrors of one skill lint once, and an
+    # entry with no digest is never absorbed into another.
+    skills = content_unique(kept)
 
     name_counts: dict[str, int] = {}
     for entry in skills:

{boost_skill_cli-1.2.151 → boost_skill_cli-1.2.153}/boost_skill_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.2.151
+Version: 1.2.153
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License-Expression: Apache-2.0

{boost_skill_cli-1.2.151 → boost_skill_cli-1.2.153}/boost_skill_cli.egg-info/SOURCES.txt

@@ -903,6 +903,7 @@ tests/unit/test_builtin.py
 tests/unit/test_canary_deps.py
 tests/unit/test_capabilities.py
 tests/unit/test_catalog.py
+tests/unit/test_catalog_distinct.py
 tests/unit/test_catalog_keyword_words.py
 tests/unit/test_catalogbundle.py
 tests/unit/test_chat.py

{boost_skill_cli-1.2.151 → boost_skill_cli-1.2.153}/docs/roadmap/items/eval-corpus-counted-by-the-method-the-repo-disavows.md

@@ -1,19 +1,34 @@
 ---
 id: eval-corpus-counted-by-the-method-the-repo-disavows
 board: code
-section: planned
-status: planned
+section: shipped
+status: shipped
 category: Quality · Retrieval eval
 complexity: L
 impact: High
 wow: 4
-note: scripts/eval_corpus.py counts a repo's contribution as len(entries) from a raw scan (…
+note: taps.txt rows gained a distinct-content count and the gate gained a second ceiling. The raw count stays — BM25 indexes without de-duplicating, so the copies really are documents — but sickn33 is 61.8% of the rows and 35.7% of the content, and only one number was being floored.
 order: 208
-owner:
-pr:
+owner: loop/eval-corpus-distinct-count
+pr: 992
 title: The eval corpus's size and its concentration ceiling are counted with <code>len(scan_dir)</code> — the measure <code>measure_registry.py</code> exists to say is wrong — so 44.7% of the gate's corpus is vendored …
 ---
-<b>Measured.</b> Two tools in this repo, run on the same clone at the same pinned SHA, give item counts 3.16x apart — <code>scripts/measure_registry.py</code> says <code>est_items=2100</code> while <code>scripts/eval_corpus.py --ensure</code> records <code>6634 entries</code> — and the tool that is right is the one the eval gate does not use: measure_registry.py's own docstring (lines 6-11) states that <code>len(catalog.scan_dir(repo))</code> is not the measurement, "the same rule the eval gate's ranked list uses". <code>eval_corpus.py:299</code> is still <code>counts[repo] = len(entries)</code>, and <code>MAX_SHARE = 0.65</code> (line 113) ratchets on that number inside CI's required <code>lint</code> job — so 976 extra vendored copies from one stranger's repository, containing zero new content, turn every open pull request red.
+<b>SHIPPED.</b> taps.txt rows gained a fourth field — the distinct-content count — and the concentration gate became two ceilings instead of one.
+
+<code>anthropics/skills   3b3fad9…   20    20</code><br>
+<code>sickn33/antigravity-awesome-skills   782f684…   6634  2117</code>
+
+<code>MAX_SHARE</code> is now <code>MAX_ROW_SHARE</code> (0.65) and is joined by <code>MAX_CONTENT_SHARE</code> (0.40), and both are checked on every <code>--ensure</code>, <code>--relock</code>, <code>--refresh</code> and <code>--audit</code>. At the current pins: sickn33 is <b>61.8% of the rows and 35.7% of the content</b>, and affaan-m/ECC is <b>15.1% of rows and 26.7% of content</b> — the two orderings differ, which is the whole reason there are two numbers. The corpus is 10,731 rows over 5,938 distinct items, so the rows overstate it <b>1.81x</b>, and taps.txt now says so in its own generated size block.
+
+<b>The raw count was kept, deliberately.</b> This card's title reads as "the raw count is wrong", and the fix is not to replace it. <code>rag.build</code> indexes <code>catalog.all_entries()</code> with <b>no</b> de-duplication, so a registry vendoring the same skill sixty-eight times really does occupy sixty-eight of the documents BM25 ranks a query against: the raw count is the honest number for the DRIFT check and for "BM25 scored N documents". What was missing was the second number, not a correction to the first. Hence two fields and two ceilings rather than one replaced.
+
+<b>The identity is <code>catalog._content_digest</code>, not <code>measure_registry.py</code>'s.</b> Two new helpers in <code>boost_cli/core/catalog.py</code> — <code>content_unique()</code> and <code>distinct_content()</code> — collapse on <code>entry["content"]</code>, the digest over name + description + body already stamped at scan time and already what <code>rag.dedupe_by_content</code> uses. <code>measure_registry.py</code>'s digest hashes the body <em>alone</em> with agent-dotdir tokens normalised, which is right for <code>est_items</code> (one skill rendered into five agents' directories is one item) and wrong here (two different skills with the same body are two documents in the index). The card's premise that these are the same rule is what <code>measure_registry.py</code>'s docstring claims and is not true; the gate now uses the one that matches what it scores.
+
+<b>An unsatisfiable ceiling is not a strict one.</b> A ceiling of <em>c</em> cannot be met by fewer than <code>1/c</code> repositories, because the smallest possible top share is <code>1/N</code> — two repos cannot get below 50%, so a 40% content ceiling would refuse every two-repo corpus however it was balanced, and the only remedy it could print ("add breadth") is the only thing that could ever clear it. <code>check_concentration</code> therefore does not judge a corpus below that floor. The shipped list has twenty repos against a floor of three, so the real gate is untouched; what it buys is that a fixture corpus can exercise the same code path.
+
+<b>Not regenerated: <code>tests/eval/baseline.json</code>.</b> No pin moved and the indexed corpus is byte-identical, so the scores are unchanged — regenerating it would have been noise in the diff and a claim that something had been re-measured.
+
+<b>Measured.</b> Two tools in this repo, run on the same clone at the same pinned SHA, give item counts 3.16x apart — <code>scripts/measure_registry.py</code> says <code>est_items=2100</code> while <code>scripts/eval_corpus.py --ensure</code> records <code>6634 entries</code> — and the tool that is right is the one the eval gate does not use: measure_registry.py's own docstring (lines 6-11) states that <code>len(catalog.scan_dir(repo))</code> is not the measurement, "the same rule the eval gate's ranked list uses". <code>eval_corpus.py</code> was still <code>counts[repo] = len(entries)</code>, and <code>MAX_SHARE = 0.65</code> ratcheted on that number inside CI's required <code>lint</code> job — so 975 extra vendored copies from one stranger's repository, containing zero new content, turn every open pull request red.
 
 <b>Reproduce it.</b>
 
@@ -33,7 +48,7 @@ title: The eval corpus's size and its concentration ceiling are counted with <co
 
 <b>What adversarial verification corrected.</b> This card's numbers are the re-measured ones, not the ones first reported.
 
-1. MATERIALLY WRONG — "The vendoring is 60 <code>plugins/agentic-bundle-*/</code> directories". There are 58 <code>agentic-bundle-*</code> dirs (60 <code>plugins/*</code> dirs total), and they hold only 449 of the 6,634 entries (6.8%; median 8 entries each). The 3.13x inflation is two FULL CATALOG MIRRORS: <code>plugins/agentic-awesome-skills-claude</code> (2,049 entries) and <code>plugins/agentic-awesome-skills</code> (2,027) = 4,076 entries, digest-identical to <code>skills/</code> (2,107 entries, all distinct). This reframes the ceiling scenario: "one more bundle render... adds ~2,117 entries" is impossible — a bundle is ~8-10 entries. The correct unit is "one more per-agent mirror", ~2,049 entries -&gt; 67.9%, still over 65%. Sharper still: only 976 extra vendored sickn33 entries are needed to cross the ceiling (&lt; half a mirror). The conclusion survives; the unit does not — and the correct unit is exactly the pattern the <code>est-items</code> card already documents.
+1. MATERIALLY WRONG — "The vendoring is 60 <code>plugins/agentic-bundle-*/</code> directories". There are 58 <code>agentic-bundle-*</code> dirs (60 <code>plugins/*</code> dirs total), and they hold only 449 of the 6,634 entries (6.8%; median 8 entries each). The 3.13x inflation is two FULL CATALOG MIRRORS: <code>plugins/agentic-awesome-skills-claude</code> (2,049 entries) and <code>plugins/agentic-awesome-skills</code> (2,027) = 4,076 entries, digest-identical to <code>skills/</code> (2,107 entries, all distinct). This reframes the ceiling scenario: "one more bundle render... adds ~2,117 entries" is impossible — a bundle is ~8-10 entries. The correct unit is "one more per-agent mirror", ~2,049 entries -&gt; 67.9%, still over 65%. Sharper still: only 975 extra vendored sickn33 entries are needed to cross the ceiling (&lt; half a mirror) — 6634+975 of 10731+975 is 65.001%, and 974 is 64.998%. The conclusion survives; the unit does not — and the correct unit is exactly the pattern the <code>est-items</code> card already documents.
 
 2. "Read all 318 titles from <code>grep -h '^title:' docs/roadmap/items/*.md</code>" — there are 432 items / 432 title lines today (verified identical to origin/main except one unrelated file). The not-carded sweep was run over a smaller set than exists.
 

{boost_skill_cli-1.2.151 → boost_skill_cli-1.2.153}/docs/roadmap/items/eval-floor-calibration-stale-after-pin-refresh.md

@@ -1,16 +1,16 @@
 ---
 id: eval-floor-calibration-stale-after-pin-refresh
 board: code
-section: planned
-status: planned
+section: shipped
+status: shipped
 category: Quality · Retrieval eval
 complexity: M
 impact: Med
 wow: 3
 note: CLAUDE.md states the gate's four floors sit "~10% under" measured values of 0.852 / 0…
 order: 210
-owner:
-pr:
+owner: loop/eval-floor-calibration
+pr: "993"
 title: The 2026-09-01 pin refresh moved the corpus and re-baselined it, but nothing re-derives the floors — CLAUDE.md's "~10% under measured" is now 7.2%–17.3%
 ---
 <b>Measured.</b> The required gate's recall floor has 5.52 queries of headroom out of 91 ((0.8407 - 0.78) x 91), against the 6.55 that CLAUDE.md's published 0.852 implies — one golden query of margin the documentation says exists and does not, because commit cbc0a58b re-baselined the corpus to 10,731 entries and left CLAUDE.md, taps.txt and eval_corpus.py all stating 10,152 / 0.852.
@@ -32,7 +32,7 @@ title: The 2026-09-01 pin refresh moved the corpus and re-baselined it, but noth
 
 <b>What adversarial verification corrected.</b> This card's numbers are the re-measured ones, not the ones first reported.
 
-1. CAUSAL FRAMING IS WRONG (the headline correction). The title and claim say the 2026-09-01 pin refresh is what made "~10% under" untrue. It was untrue the day it was written. <code>git log -S'~10% under'</code> traces the sentence to 170d52c0 (2026-07-31), whose own commit message says the floors were "re-derived against the 20-tap numbers at the same ~10% relative headroom" against measured 0.863 / 0.473 / 0.607 / 0.662 — at which point the gaps were 9.6% / 15.4% / 14.3% / 12.4%, a 1.60x spread. The refresh WIDENED a spread (1.60x -&gt; 1.82x -&gt; 2.39x); it did not create one. Correct statement: "the floors were never uniformly ~10% under, and the refresh widened the spread from 1.82x to 2.39x."
+1. CAUSAL FRAMING IS WRONG (the headline correction). The title and claim say the 2026-09-01 pin refresh is what made "~10% under" untrue. It was untrue the day it was written. <code>git log -S'~10% under'</code> traces the sentence to 170d52c0 (2026-07-31), whose own commit message says the floors were "re-derived against the 20-tap numbers at the same ~10% relative headroom" against measured 0.863 / 0.473 / 0.607 / 0.662 — at which point the gaps were 9.6% / 15.3% / 14.4% / 12.4%, a 1.60x spread. (This line first read 9.6% / 15.4% / 14.3% / 12.4%; recomputing from 170d52c0's own <code>baseline.json</code> and <code>Makefile</code> gives 9.5796 / 15.3488 / 14.3831 / 12.4408, so two digits were transposed.) The spread widened twice and by two different mechanisms, only one of which is a refresh: 1.60x at 170d52c0, 1.82x at d8da2e71 — <em>de-duplicate the ranked list by body, not by name</em>, a ranker correctness fix (#411) — and 2.39x at the 2026-09-01 refresh cbc0a58b. Correct statement: "the floors were never uniformly ~10% under, and nothing since has narrowed the spread."
 
 2. "not the 6.60 the documented number implies" — 6.60 matches nothing. CLAUDE.md's published 0.852 gives (0.852-0.78) x 91 = 6.55 queries; the unrounded pre-refresh baseline 0.8406... gives 6.52. Use 6.55. (The optimism is therefore ~1.03 queries, not 1.1.)
 
@@ -53,3 +53,9 @@ SCOPE LIMIT #2: the shared $TMPDIR/eval-home is materialised at the PRE-refresh
 <b>Why it is worth doing.</b> The "~10% under" statement is the entire published justification for where the four floors sit — CLAUDE.md calls it "loose enough that upstream drift can't flake the build, tight enough to catch a collapse." That is a claim about a margin, and the margin now differs 2.4x across the four metrics with recall the tightest at 5.52 queries out of 91. Because the monthly refresh re-baselines but never re-derives or re-states the floors, and prompts a human only when the refreshed corpus FAILS, every future passing refresh decays the calibration further with no signal at all.
 
 <em>Found by an automated audit of retrieval/eval quality, the search &amp; browse surfaces, and first-run onboarding; every finding was then re-measured from scratch by an independent adversarial verifier whose instruction was to refute it. Verdict: <b>CORRECTED</b>. No fix is prescribed here — the measurement is the contribution.</em>
+
+<b>What the fix turned out to be, and what the card got wrong about scope.</b> Half of this card was already fixed when it was picked up. The figure half — "CLAUDE.md, taps.txt and eval_corpus.py all stating 10,152 / 0.852" — is stale: <code>tests/unit/test_corpus_prose.py</code> now holds every one of those figures to <code>taps.txt</code>'s own rows and to <code>baseline.json</code>, so CLAUDE.md reads 10,731 and 0.841 / 0.484 / 0.607 / 0.655 and cannot drift again. The card's line references (CLAUDE.md:67,84) no longer point at anything. What survived is the card's own stated core, and it is the harder half: the <em>characterisation</em> "~10% under" was prose, was false the day it was written, and no mechanism touched it.
+
+<b>The remedy is the card's own, not the one the finding's wording invites.</b> No floor moves and no headroom gate is added — a floor that follows its own measurement ends up measuring the corpus, which is exactly what 170d52c0 warned about. Instead the true margin is written down at measured precision (7.2% / 17.3% / 14.3% / 11.5%, a 2.39x spread, recall@k binding at 5.5 queries of 91) and held by the <code>QUOTED</code> mechanism that already holds the figures beside it. A refresh that moves the row now fails the build until a human restates the margin and looks at it — which is the signal the card says is missing, without touching the gate it says not to touch.
+
+<b>Why the required set drifted and the advisory set did not.</b> The contrast is the tell. <code>make eval-natural</code>'s floors really are ~10% under (11.1 / 12.5 / 11.4 / 11.2, a 1.12x spread) because <code>tests/unit/test_eval_corpus.py</code> moves them with the row in both directions. The required set, where the claim actually matters, had no such mechanism — so the one place the margin was load-bearing is the one place nothing checked it.

{boost_skill_cli-1.2.151 → boost_skill_cli-1.2.153}/docs/roadmap/items/eval-gate-scores-a-body-less-corpus.md

@@ -52,6 +52,6 @@ WHY Med, NOT High — two facts I verified that cap the blast radius: 1. CI cann
 
 SCOPE LIMITS OF MY REPRO — a card author must not blur these: - I measured the FULLY-missing case. The "half-materialised tap tree" phrasing in <code>why_it_matters</code> was NOT measured by me; degradation is presumably proportional but is unverified.
 
-<b>Why it is worth doing.</b> The gate whose whole job is to say "retrieval still works over a real corpus" returns a confident, better-than-baseline PASS over a corpus that is name+description only — 95% of its text absent. A developer who reclaims the 265 MB <code>repos/</code> tree, or whose tap tree is half-materialised, gets four green floors that attest to nothing, and the greener number makes the loss look like an improvement rather than a defect. Every downstream floor calibration ("each floor sits ~10% under its measured value") is stated against 0.473/0.605/0.657, which the body-less run clears by 48%/33%/25% instead.
+<b>Why it is worth doing.</b> The gate whose whole job is to say "retrieval still works over a real corpus" returns a confident, better-than-baseline PASS over a corpus that is name+description only — 95% of its text absent. A developer who reclaims the 265 MB <code>repos/</code> tree, or whose tap tree is half-materialised, gets four green floors that attest to nothing, and the greener number makes the loss look like an improvement rather than a defect. Every downstream floor calibration is stated against 0.473/0.605/0.657, which the body-less run clears by 48%/33%/25% instead. (This line quoted CLAUDE.md's "each floor sits ~10% under its measured value" as the live convention. That sentence was retired in PR #993: the margins were never uniform — 7.2% / 17.3% / 14.3% / 11.5% at the current pins — and CLAUDE.md now quotes all four and holds them.)
 
 <em>Found by an automated audit of retrieval/eval quality, the search &amp; browse surfaces, and first-run onboarding; every finding was then re-measured from scratch by an independent adversarial verifier whose instruction was to refute it. Verdict: <b>CORRECTED</b>. No fix is prescribed here — the measurement is the contribution.</em>

{boost_skill_cli-1.2.151 → boost_skill_cli-1.2.153}/docs/roadmap.html

@@ -140,9 +140,9 @@
   <section class="block reveal">
     <div class="stats">
       <!-- ROADMAP:stats -->
-      <div class="stat"><b>23</b><span>Shipped</span></div>
+      <div class="stat"><b>25</b><span>Shipped</span></div>
       <div class="stat"><b>1</b><span>Next up</span></div>
-      <div class="stat"><b>78</b><span>Planned</span></div>
+      <div class="stat"><b>76</b><span>Planned</span></div>
       <div class="stat"><b>505</b><span>Loop finds</span></div>
       <!-- /ROADMAP:stats -->
     </div>
@@ -408,6 +408,30 @@
         </div>
       </article>
 
+      <article class="cap rcard" id="eval-corpus-counted-by-the-method-the-repo-disavows">
+        <div class="head"><span class="pill shipped">Shipped</span><span class="cat">Quality · Retrieval eval</span></div>
+        <h3>The eval corpus's size and its concentration ceiling are counted with <code>len(scan_dir)</code> — the measure <code>measure_registry.py</code> exists to say is wrong — so 44.7% of the gate's corpus is vendored …</h3>
+        <p class="cardbody"><a class="writeup" href="https://github.com/jonnyeclectic/boost/blob/main/docs/roadmap/items/eval-corpus-counted-by-the-method-the-repo-disavows.md">Write-up &middot; eval-corpus-counted-by-the-method-the-repo-disavows.md</a></p>
+        <div class="meta">
+          <span class="m">Complexity <b>L</b></span>
+          <span class="m hi">Impact <b>High</b></span>
+          <span class="m">Wow <b class="wow">★★★★</b></span>
+          <span class="m">taps.txt rows gained a distinct-content count and the gate gained a second ceiling. The raw count stays — BM25 indexes without de-duplicating, so the copies really are documents — but sickn33 is 61.8% of the rows and 35.7% of the content, and only one number was being floored.</span>
+        </div>
+      </article>
+
+      <article class="cap rcard" id="eval-floor-calibration-stale-after-pin-refresh">
+        <div class="head"><span class="pill shipped">Shipped</span><span class="cat">Quality · Retrieval eval</span></div>
+        <h3>The 2026-09-01 pin refresh moved the corpus and re-baselined it, but nothing re-derives the floors — CLAUDE.md's "~10% under measured" is now 7.2%–17.3%</h3>
+        <p class="cardbody"><a class="writeup" href="https://github.com/jonnyeclectic/boost/blob/main/docs/roadmap/items/eval-floor-calibration-stale-after-pin-refresh.md">Write-up &middot; eval-floor-calibration-stale-after-pin-refresh.md</a></p>
+        <div class="meta">
+          <span class="m">Complexity <b>M</b></span>
+          <span class="m">Impact <b>Med</b></span>
+          <span class="m">Wow <b class="wow">★★★</b></span>
+          <span class="m">CLAUDE.md states the gate's four floors sit "~10% under" measured values of 0.852 / 0…</span>
+        </div>
+      </article>
+
       <article class="cap rcard" id="install-cannot-tell-a-skill-needs-another-one">
         <div class="head"><span class="pill shipped">Shipped</span><span class="cat">Install · Research</span></div>
         <h3>Research: how <code>boost install</code> could tell that a skill needs something else installed first</h3>
@@ -1148,52 +1172,6 @@ space before trusting a number in it.</p>
         </div>
       </article>
 
-      <article class="cap rcard" id="eval-corpus-counted-by-the-method-the-repo-disavows">
-        <div class="head"><span class="pill planned">Planned</span><span class="cat">Quality · Retrieval eval</span></div>
-        <h3>The eval corpus's size and its concentration ceiling are counted with <code>len(scan_dir)</code> — the measure <code>measure_registry.py</code> exists to say is wrong — so 44.7% of the gate's corpus is vendored …</h3>
-        <p><b>Measured.</b> Two tools in this repo, run on the same clone at the same pinned SHA, give item counts 3.16x apart — <code>scripts/measure_registry.py</code> says <code>est_items=2100</code> while <code>scripts/eval_corpus.py --ensure</code> records <code>6634 entries</code> — and the tool that is right is the one the eval gate does not use: measure_registry.py's own docstring (lines 6-11) states that <code>len(catalog.scan_dir(repo))</code> is not the measurement, "the same rule the eval gate's ranked list uses". <code>eval_corpus.py:299</code> is still <code>counts[repo] = len(entries)</code>, and <code>MAX_SHARE = 0.65</code> (line 113) ratchets on that number inside CI's required <code>lint</code> job — so 976 extra vendored copies from one stranger's repository, containing zero new content, turn every open pull request red.
-
-<b>Reproduce it.</b>
-
-<code>cd &lt;repo&gt;</code><br>
-<code>export HOME=$TMPDIR/audit-corpus-verify &amp;&amp; mkdir -p "$HOME" &amp;&amp; export BOOST_HOME=$HOME/.boost</code><br>
-<code>.venv/bin/python scripts/eval_corpus.py --ensure      # -&gt; "corpus: 10731 entries"; "concentration: ... 61.8%"</code><br>
-<code>.venv/bin/python scripts/measure_registry.py "$BOOST_HOME/repos/sickn33__antigravity-awesome-skills"   # -&gt; est_items=2100</code><br>
-<code>.venv/bin/python - &lt;&lt;'PY'</code><br>
-<code>import json, pathlib, collections, os</code><br>
-<code>home=pathlib.Path(os.environ["BOOST_HOME"]); ents=[]</code><br>
-<code>for f in sorted((home/"cache").glob("*.json")):</code><br>
-<code>    if f.stem.startswith("rag_"): continue</code><br>
-<code>    d=json.loads(f.read_text())</code><br>
-<code>    for e in d["skills"]: e["_tap"]=d["tap"]; ents.append(e)</code><br>
-<code>alld={e["content"] for e in ents}</code><br>
-…
-
-<b>What adversarial verification corrected.</b> This card's numbers are the re-measured ones, not the ones first reported.
-
-1. MATERIALLY WRONG — "The vendoring is 60 <code>plugins/agentic-bundle-*/</code> directories". There are 58 <code>agentic-bundle-*</code> dirs (60 <code>plugins/*</code> dirs total), and they hold only 449 of the 6,634 entries (6.8%; median 8 entries each). The 3.13x inflation is two FULL CATALOG MIRRORS: <code>plugins/agentic-awesome-skills-claude</code> (2,049 entries) and <code>plugins/agentic-awesome-skills</code> (2,027) = 4,076 entries, digest-identical to <code>skills/</code> (2,107 entries, all distinct). This reframes the ceiling scenario: "one more bundle render... adds ~2,117 entries" is impossible — a bundle is ~8-10 entries. The correct unit is "one more per-agent mirror", ~2,049 entries -&gt; 67.9%, still over 65%. Sharper still: only 976 extra vendored sickn33 entries are needed to cross the ceiling (&lt; half a mirror). The conclusion survives; the unit does not — and the correct unit is exactly the pattern the <code>est-items</code> card already documents.
-
-2. "Read all 318 titles from <code>grep -h '^title:' docs/roadmap/items/*.md</code>" — there are 432 items / 432 title lines today (verified identical to origin/main except one unrelated file). The not-carded sweep was run over a smaller set than exists.
-
-3. "Every one of them... none measures distinct content" is wrong.
-
-<b>What is NOT established.</b> Recorded because a card that overstates its own evidence is worse than no card.
-
-SCOPE OF MY REPRO — what I actually settled, and what I did not. - I materialised the corpus at the CURRENT taps.txt pins in my own disposable HOME and reproduced every headline number independently. The pre-refresh figures (10,152 / 5,790 / 62.1% / 34.5% / 3.16x) I took from the SHARED read-only eval-home at $TMPDIR/eval-home, which is still at the older pins — a read-only census, no writes. - The repo checkout was on a peer session's branch <code>loop/missing-json</code>, not <code>main</code>. I diffed every file this finding touches (scripts/eval_corpus.py, tests/eval/taps.txt, boost_cli/core/rag.py, .github/workflows/eval-corpus-refresh.yml, all of docs/roadmap/items/) against origin/main: identical except one unrelated roadmap item. The verification holds against main (origin/main af4bdbfd). - I could NOT isolate the causal mechanism of the 22 rank changes. The finding asserts "BM25 statistics (N and document frequency), not slot consumption". My data is CONSISTENT with that — <code>dedupe_by_content</code> does run over the full pool before k (rag.py:1053, confirmed), and recall@k is bit-identical while the rank-sensitive metrics move — but I did not separate idf from avgdl, nor rule out that a different byte-identical copy survives dedupe and carries a different <code>grade_key</code>. Write it as "consistent with", not "confirmed".
-
-WHY HIGH RATHER THAN MEDIUM. The metric movement alone cannot fail anything: hit@1 +0.011, MRR +0.010, nDCG +0.008 against 0.06-0.10 of headroom over the floors. The severity is the ratchet's teeth, which the finder never traced.
-
-<b>Why it is worth doing.</b> <code>MAX_SHARE</code> is the only shipped guard against the gate's corpus becoming one publisher's house style, and it is measured on a quantity a third party can inflate by ~3x without publishing a single new skill — so it can fire on a repo contributing a third of the content, and cannot fire on a repo that dominates the content without vendoring. The same raw count is the headline everywhere (<code>10,152 entries</code> / <code>62%</code>) and is what the monthly refresh PR reports as growth, overstating real content growth 4x.
-
-<em>Found by an automated audit of retrieval/eval quality, the search &amp; browse surfaces, and first-run onboarding; every finding was then re-measured from scratch by an independent adversarial verifier whose instruction was to refute it. Verdict: <b>CORRECTED</b>. No fix is prescribed here — the measurement is the contribution.</em></p>
-        <div class="meta">
-          <span class="m">Complexity <b>L</b></span>
-          <span class="m hi">Impact <b>High</b></span>
-          <span class="m">Wow <b class="wow">★★★★</b></span>
-          <span class="m">scripts/eval_corpus.py counts a repo's contribution as len(entries) from a raw scan (…</span>
-        </div>
-      </article>
-
       <article class="cap rcard" id="eval-corpus-duplication-is-intra-tap-not-mirrors">
         <div class="head"><span class="pill planned">Planned</span><span class="cat">Quality · Retrieval eval</span></div>
         <h3>The corpus's 65.6% duplication is 99.93% inside a single tap, so the required gate never once exercises the cross-tap trust ordering dedup exists for — 0 swaps in 264,735 comparisons</h3>
@@ -1242,57 +1220,6 @@ NOT ALREADY CARDED, but read the existing card first.
         </div>
       </article>
 
-      <article class="cap rcard" id="eval-floor-calibration-stale-after-pin-refresh">
-        <div class="head"><span class="pill planned">Planned</span><span class="cat">Quality · Retrieval eval</span></div>
-        <h3>The 2026-09-01 pin refresh moved the corpus and re-baselined it, but nothing re-derives the floors — CLAUDE.md's "~10% under measured" is now 7.2%–17.3%</h3>
-        <p><b>Measured.</b> The required gate's recall floor has 5.52 queries of headroom out of 91 ((0.8407 - 0.78) x 91), against the 6.55 that CLAUDE.md's published 0.852 implies — one golden query of margin the documentation says exists and does not, because commit cbc0a58b re-baselined the corpus to 10,731 entries and left CLAUDE.md, taps.txt and eval_corpus.py all stating 10,152 / 0.852.
-
-<b>Reproduce it.</b>
-
-<code>cd &lt;repo&gt;</code><br>
-<code># 1. the refresh commit moved pins + baseline and nothing else</code><br>
-<code>git show cbc0a58b --stat --format='%h %ad %s' --date=short</code><br>
-<code>git show cbc0a58b -- tests/eval/baseline.json | grep -E '^[-+].*(recall|hit@1|MRR|nDCG)'</code><br>
-<code># 2. what taps.txt records NOW vs what the docs claim</code><br>
-<code>awk '!/^#/ &amp;&amp; NF&gt;=3 {s+=$3; n++} END {print "rows:", n, " recorded total entries:", s}' tests/eval/taps.txt</code><br>
-<code>grep -n '10,152\|0\.852' CLAUDE.md tests/eval/taps.txt scripts/eval_corpus.py</code><br>
-<code># 3. recompute the floor gaps against the committed post-refresh baseline</code><br>
-<code>python3 -c "</code><br>
-<code>import json</code><br>
-<code>b=json.load(open('tests/eval/baseline.json'))['sets']['golden.jsonl@a0617183f8c9']['engines']['BM25 full-content']</code><br>
-…
-
-<b>What adversarial verification corrected.</b> This card's numbers are the re-measured ones, not the ones first reported.
-
-1. CAUSAL FRAMING IS WRONG (the headline correction). The title and claim say the 2026-09-01 pin refresh is what made "~10% under" untrue. It was untrue the day it was written. <code>git log -S'~10% under'</code> traces the sentence to 170d52c0 (2026-07-31), whose own commit message says the floors were "re-derived against the 20-tap numbers at the same ~10% relative headroom" against measured 0.863 / 0.473 / 0.607 / 0.662 — at which point the gaps were 9.6% / 15.4% / 14.3% / 12.4%, a 1.60x spread. The refresh WIDENED a spread (1.60x -&gt; 1.82x -&gt; 2.39x); it did not create one. Correct statement: "the floors were never uniformly ~10% under, and the refresh widened the spread from 1.82x to 2.39x."
-
-2. "not the 6.60 the documented number implies" — 6.60 matches nothing. CLAUDE.md's published 0.852 gives (0.852-0.78) x 91 = 6.55 queries; the unrounded pre-refresh baseline 0.8406... gives 6.52. Use 6.55. (The optimism is therefore ~1.03 queries, not 1.1.)
-
-3. "2.4x spread" -&gt; 2.39x (rounding, harmless, but state it as 2.39x on a card).
-
-4. DIRECTION OF DRIFT IS NOT UNIFORM. "every future passing refresh decays the calibration further" is true only of the SPREAD. This refresh moved two floors tighter (recall 8.4%-&gt;7.2%, nDCG 11.8%-&gt;11.5%) and two LOOSER (hit@1 15.3%-&gt;17.3%, MRR 14.0%-&gt;14.3%). A card must not say "the floors tightened."
-
-5.
-
-<b>What is NOT established.</b> Recorded because a card that overstates its own evidence is worse than no card.
-
-SCOPE THE REMEDY CAREFULLY — the finding's phrase "nothing re-derives the floors" invites exactly the fix 170d52c0 warns against: auto-moving floors on every refresh ("Lowering a threshold deserves suspicion... a floor calibrated on an unrepresentative corpus measures the corpus"). The repo's design is deliberately fixed absolute floors with a relaxed regression-vs-baseline. The CONFIRMED defect is narrower and should be carded as such: the measured values and the "~10%" characterisation are stated in seven places across four files (CLAUDE.md:67,84; tests/eval/taps.txt:25,43; scripts/eval_corpus.py:30,108 and the runtime-printed operator message at :331-334), nothing re-states or guards them after a PASSING refresh, and no test asserts any of them.
-
-SCOPE LIMIT OF MY REPRO #1: I did not re-measure 0.8407 / 0.4835 / 0.6065 / 0.6552. Those are the committed baseline written by cbc0a58b — which is also the finder's only source. Verifying them empirically needs a re-tap at the new pins, which is forbidden here (shared read-only corpus) and needs network. Every derived percentage in this verification inherits that.
-
-SCOPE LIMIT #2: the shared $TMPDIR/eval-home is materialised at the PRE-refresh pins — live total 10,152 vs taps.txt's recorded 10,731, with the same 10 taps PIN-DIFFERS. <code>make eval</code> against it would trip eval_corpus.py's CORPUS DRIFT check, not run the gate.
-
-<b>Why it is worth doing.</b> The "~10% under" statement is the entire published justification for where the four floors sit — CLAUDE.md calls it "loose enough that upstream drift can't flake the build, tight enough to catch a collapse." That is a claim about a margin, and the margin now differs 2.4x across the four metrics with recall the tightest at 5.52 queries out of 91. Because the monthly refresh re-baselines but never re-derives or re-states the floors, and prompts a human only when the refreshed corpus FAILS, every future passing refresh decays the calibration further with no signal at all.
-
-<em>Found by an automated audit of retrieval/eval quality, the search &amp; browse surfaces, and first-run onboarding; every finding was then re-measured from scratch by an independent adversarial verifier whose instruction was to refute it. Verdict: <b>CORRECTED</b>. No fix is prescribed here — the measurement is the contribution.</em></p>
-        <div class="meta">
-          <span class="m">Complexity <b>M</b></span>
-          <span class="m">Impact <b>Med</b></span>
-          <span class="m">Wow <b class="wow">★★★</b></span>
-          <span class="m">CLAUDE.md states the gate's four floors sit "~10% under" measured values of 0.852 / 0…</span>
-        </div>
-      </article>
-
       <article class="cap rcard" id="eval-gate-scores-a-body-less-corpus">
         <div class="head"><span class="pill shipped">Shipped</span><span class="cat">Quality · Retrieval eval</span></div>
         <h3><code>make eval</code> scores a corpus with every SKILL.md body missing, reports all four floors PASS, and scores HIGHER than the real corpus</h3>