boost-skill-cli 1.0.358__tar.gz → 1.0.359__tar.gz

{boost_skill_cli-1.0.358 → boost_skill_cli-1.0.359}/CLAUDE.md

@@ -63,12 +63,22 @@ corpus: `scripts/ensure_eval_corpus.sh` first taps the pinned repo list in
 --defaults` is NOT enough, it omits every rule/workflow repo). The list is
 **twenty** repos: the first six cover every golden target, the rest exist so the
 corpus is a realistic size. That matters more than it sounds — over the six
-alone BM25 scored 1.000 / 0.791 / 0.854 / 0.882, and over the twenty it scores
-**0.863 / 0.473 / 0.607 / 0.662** on the same golden set, so three of the four
-old floors fail once the corpus stops being tiny. Each floor now sits ~10% under
-its measured value — loose enough that upstream drift can't flake the build,
-tight enough to catch a collapse. Regression-vs-baseline stays relaxed
-(`--regression-eps 1`), so the absolute floors are the real gate.
+alone (743 entries) BM25 scores 0.978 / 0.791 / 0.854 / 0.882, and over the
+twenty (10,152) it scores **0.852 / 0.473 / 0.605 / 0.657** on the same golden
+set, so three of the four old floors fail once the corpus stops being tiny.
+
+**The ranked list de-duplicates on the content hash, not the name.** A grade key
+decides both relevance and identity, and keying identity on the name collapsed
+13 different skills called `code-reviewer` into one rank slot — crediting the
+ranker with a compression that existed only in the scoring code, and worth about
+one query of recall@10. That is where the old "recall is 1.000" folklore came
+from; the six-repo corpus measures 0.978 once mirrors collapse and homonyms do
+not. Relevance is still decided by name (or by content class when a golden row
+pins an `exemplar`), so the sets can migrate a row at a time.
+
+Each floor sits ~10% under its measured value — loose enough that upstream drift
+can't flake the build, tight enough to catch a collapse. Regression-vs-baseline
+stays relaxed (`--regression-eps 1`), so the absolute floors are the real gate.
 
 **Every row of `taps.txt` pins a commit SHA**, and `scripts/eval_corpus.py`
 checks each clone out at it — the corpus is 10,152 entries, of which one

{boost_skill_cli-1.0.358 → boost_skill_cli-1.0.359}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.0.358
+Version: 1.0.359
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License: GNU GENERAL PUBLIC LICENSE

{boost_skill_cli-1.0.358 → boost_skill_cli-1.0.359}/boost_cli/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '1.0.358'
-__version_tuple__ = version_tuple = (1, 0, 358)
+__version__ = version = '1.0.359'
+__version_tuple__ = version_tuple = (1, 0, 359)
 
 __commit_id__ = commit_id = None

{boost_skill_cli-1.0.358 → boost_skill_cli-1.0.359}/boost_skill_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.0.358
+Version: 1.0.359
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License: GNU GENERAL PUBLIC LICENSE

{boost_skill_cli-1.0.358 → boost_skill_cli-1.0.359}/boost_skill_cli.egg-info/SOURCES.txt

@@ -214,6 +214,7 @@ docs/roadmap/items/end-of-options-guard-on-git-commands.md
 docs/roadmap/items/eval-corpus-cannot-see-real-retrieval.md
 docs/roadmap/items/eval-corpus-is-96x-smaller-than-a-real-install.md
 docs/roadmap/items/eval-corpus-was-not-actually-pinned.md
+docs/roadmap/items/eval-deduped-ranked-lists-by-name.md
 docs/roadmap/items/extension-free-tests-core-dense-py.md
 docs/roadmap/items/extract-mcp-http-servers-out-of-configuration-py.md
 docs/roadmap/items/finish-mutation-hardening-across-core.md

boost_skill_cli-1.0.359/docs/roadmap/items/eval-deduped-ranked-lists-by-name.md

@@ -0,0 +1,50 @@
+---
+id: eval-deduped-ranked-lists-by-name
+board: code
+section: internals
+status: shipped
+category: Eval · Correctness
+complexity: M
+impact: High
+wow: 4
+note: 13 different skills named code-reviewer shared one rank slot — where "recall is 1.000" came from
+order: 83
+owner: loop/eval-dedup-by-body
+pr: 411
+title: The eval de-duplicated its ranked list by name, so homonyms shared a rank
+---
+<b>A grade key does two jobs, and they needed different answers.</b> It decides whether an entry is
+<em>relevant</em>, and it is the <em>identity</em> the ranked list is de-duplicated on. Both were the
+entry's name &mdash; so the thirteen genuinely different skills called <code>code-reviewer</code> in
+the pinned corpus collapsed into <b>one rank slot</b>, and every metric was computed over a list
+about a third shorter than the one a user would scroll.
+
+<b>This is not a rounding error dressed up.</b> It credited the ranker with a compression that exists
+only in the scoring code. Measured over the pinned corpus, de-duplicating on the content hash instead
+moves BM25 recall@10 from <b>0.863 to 0.852</b> &mdash; roughly one golden query &mdash; and over the
+six-repo minimal set from <b>1.000 to 0.978</b>. That second number is the interesting one: it is
+where the project's &ldquo;retrieval recall is 1.000&rdquo; folklore came from. The corpus never had
+perfect recall; it had a scoring key that merged wrong answers into right ones.
+
+<b>Exemplar rows had the mirror-image bug.</b> When a row pins an exemplar, its <em>distractors</em>
+were keyed on <code>tap::skill_md</code> &mdash; so two byte-identical mirrors of a distractor each
+took a slot and pushed the target later. The two bugs pointed opposite ways, which meant an
+exemplar-graded row and a name-graded row could not honestly be averaged into a single number. That
+blocked finishing [[golden-set-grades-by-name-not-by-skill]], whose whole design is that rows migrate
+one at a time.
+
+<b>The fix separates the two jobs.</b> Relevance is decided by name, or by content class when the row
+pins an exemplar &mdash; unchanged, so a multi-name <code>relevant</code> list still needs each
+distinct name found. Identity is always the entry's content hash: mirrors of one skill collapse
+(counting them twice rewards nothing), distinct bodies do not (a user really does see thirteen
+entries). One convention for every row.
+
+<b>What this does not fix.</b> The corpus is still 10,152 entries against a real install's far
+larger one &mdash; see [[eval-corpus-is-96x-smaller-than-a-real-install]] &mdash; and the golden set
+still grades 22 of its 50 natural-language rows against a name that resolves to several different
+skills. This makes finishing that migration possible; it does not finish it. The corrected floors are
+comfortably clear of the gate (recall 0.852 against 0.78), so no threshold moved.
+
+<b>Provenance.</b> Found while working out how to add the remaining exemplars: the blocker turned out
+not to be the 22 judgment calls but the fact that a half-migrated set would average two different
+rank conventions. Measuring that is what exposed the name-collapse underneath it.

{boost_skill_cli-1.0.358 → boost_skill_cli-1.0.359}/docs/roadmap.html

@@ -125,7 +125,7 @@
       <div class="stat"><b>16</b><span>Shipped</span></div>
       <div class="stat"><b>1</b><span>Next up</span></div>
       <div class="stat"><b>4</b><span>Planned</span></div>
-      <div class="stat"><b>197</b><span>Loop finds</span></div>
+      <div class="stat"><b>198</b><span>Loop finds</span></div>
       <!-- /ROADMAP:stats -->
     </div>
   </section>
@@ -2485,6 +2485,53 @@ measured on an install that was missing the repo holding 62% of it.</p></details
           <span class="m">62% of the required gate's corpus was one unpinned third-party repo, against a 1.15-query margin</span>
         </div>
       </article>
+
+      <article class="cap rcard" id="eval-deduped-ranked-lists-by-name">
+        <div class="head"><span class="pill shipped">Shipped</span><span class="cat">Eval · Correctness</span></div>
+        <h3>The eval de-duplicated its ranked list by name, so homonyms shared a rank</h3>
+        <details class="cardbody"><summary>Write-up</summary>
+        <p><b>A grade key does two jobs, and they needed different answers.</b> It decides whether an entry is
+<em>relevant</em>, and it is the <em>identity</em> the ranked list is de-duplicated on. Both were the
+entry's name &mdash; so the thirteen genuinely different skills called <code>code-reviewer</code> in
+the pinned corpus collapsed into <b>one rank slot</b>, and every metric was computed over a list
+about a third shorter than the one a user would scroll.
+
+<b>This is not a rounding error dressed up.</b> It credited the ranker with a compression that exists
+only in the scoring code. Measured over the pinned corpus, de-duplicating on the content hash instead
+moves BM25 recall@10 from <b>0.863 to 0.852</b> &mdash; roughly one golden query &mdash; and over the
+six-repo minimal set from <b>1.000 to 0.978</b>. That second number is the interesting one: it is
+where the project's &ldquo;retrieval recall is 1.000&rdquo; folklore came from. The corpus never had
+perfect recall; it had a scoring key that merged wrong answers into right ones.
+
+<b>Exemplar rows had the mirror-image bug.</b> When a row pins an exemplar, its <em>distractors</em>
+were keyed on <code>tap::skill_md</code> &mdash; so two byte-identical mirrors of a distractor each
+took a slot and pushed the target later. The two bugs pointed opposite ways, which meant an
+exemplar-graded row and a name-graded row could not honestly be averaged into a single number. That
+blocked finishing [[golden-set-grades-by-name-not-by-skill]], whose whole design is that rows migrate
+one at a time.
+
+<b>The fix separates the two jobs.</b> Relevance is decided by name, or by content class when the row
+pins an exemplar &mdash; unchanged, so a multi-name <code>relevant</code> list still needs each
+distinct name found. Identity is always the entry's content hash: mirrors of one skill collapse
+(counting them twice rewards nothing), distinct bodies do not (a user really does see thirteen
+entries). One convention for every row.
+
+<b>What this does not fix.</b> The corpus is still 10,152 entries against a real install's far
+larger one &mdash; see [[eval-corpus-is-96x-smaller-than-a-real-install]] &mdash; and the golden set
+still grades 22 of its 50 natural-language rows against a name that resolves to several different
+skills. This makes finishing that migration possible; it does not finish it. The corrected floors are
+comfortably clear of the gate (recall 0.852 against 0.78), so no threshold moved.
+
+<b>Provenance.</b> Found while working out how to add the remaining exemplars: the blocker turned out
+not to be the 22 judgment calls but the fact that a half-migrated set would average two different
+rank conventions. Measuring that is what exposed the name-collapse underneath it.</p></details>
+        <div class="meta">
+          <span class="m">Complexity <b>M</b></span>
+          <span class="m hi">Impact <b>High</b></span>
+          <span class="m">Wow <b class="wow">★★★★</b></span>
+          <span class="m">13 different skills named code-reviewer shared one rank slot — where "recall is 1.000" came from</span>
+        </div>
+      </article>
       <!-- /ROADMAP:cards -->
 
     </div>

{boost_skill_cli-1.0.358 → boost_skill_cli-1.0.359}/scripts/eval_retrieval.py

@@ -126,8 +126,14 @@ METRICS: Dict[str, Callable[[Sequence[str], set, int], float]] = {
 # punish a correct answer for arriving from a mirror), while a different skill
 # sharing the name does not.
 #
-# Rows without an exemplar keep name grading byte-for-byte, so published
-# baselines stay comparable and the two styles can coexist during a migration.
+# Rows without an exemplar still decide RELEVANCE by name, so the sets can
+# migrate a row at a time. What is no longer name-keyed is IDENTITY: the ranked
+# list de-duplicates on the content hash for every row, exemplar or not. Keying
+# both on the name collapsed 13 different `code-reviewer`s into one rank slot
+# and inflated recall@10 by about one query (0.863 -> 0.852 over the pinned
+# corpus); keying an exemplar row's distractors on tap::skill_md did the
+# opposite, giving byte-identical mirrors a slot each. Mixed sets could not be
+# averaged into one number until both used the same convention.
 
 _EXEMPLAR_SEP = "::"
 
@@ -165,19 +171,39 @@ def prepare_row(row: dict, hashes: dict) -> dict:
 def grade_key(row: dict, entry: dict, hashes: dict) -> str:
     """The token this entry contributes to a ranked list, for scoring.
 
+    The key does two jobs, and they need different answers. It decides whether
+    an entry is RELEVANT — by name, or by content class when the row pins an
+    exemplar — and it is the IDENTITY the ranked list is de-duplicated on.
+
+    Keying both on the name conflated them: 13 genuinely different skills named
+    `code-reviewer` collapsed into one rank slot, so the eval credited the
+    ranker with a compression that exists only here. Measured over the pinned
+    corpus that was worth about one query of recall@10 (0.863 against 0.852).
+    Exemplar rows had the inverse bug — their distractors keyed on
+    ``tap::skill_md``, so byte-identical mirrors each took a slot and pushed the
+    target later.
+
+    So: relevance by name or class, identity always by content hash. One
+    convention for every row, which is what allows an exemplar-graded row and a
+    name-graded row to be averaged into the same number.
+
     ``hashes`` is passed rather than read from module state: the map is the
     thing that decides whether two entries are the same skill, so a caller must
     not be able to grade against a different one by accident.
     """
+    digest = hashes.get((entry.get("tap", ""), entry.get("skill_md", "")))
     classes = row.get("class_hashes")
-    if not classes:
+    if classes:
+        if digest and digest in classes:
+            return "cls:%s" % sorted(classes)[0]
+    elif str(entry.get("name", "")) in row["relevant_set"]:
         return str(entry.get("name", ""))
-    digest = hashes.get((entry.get("tap", ""), entry.get("skill_md", "")))
-    if digest and digest in classes:
-        return "cls:%s" % sorted(classes)[0]
-    # Distinct, so two different homonyms never collapse into one another and
-    # inflate recall.
-    return "not:%s::%s" % (entry.get("tap", ""), entry.get("skill_md", ""))
+    if digest:
+        return "body:%s" % digest
+    # No hash (an entry the index never saw): fall back to a key that is unique
+    # per entry. Colliding here would silently shorten the ranked list and
+    # flatter every metric computed from it.
+    return "nohash:%s::%s" % (entry.get("tap", ""), entry.get("skill_md", ""))
 
 
 def relevant_keys(row: dict) -> set:

{boost_skill_cli-1.0.358 → boost_skill_cli-1.0.359}/tests/eval/baseline.json

@@ -5,16 +5,16 @@
       "golden": "golden.jsonl",
       "engines": {
         "catalog.search": {
-          "recall@k": 0.7527472527472527,
+          "recall@k": 0.7417582417582418,
           "hit@1": 0.46153846153846156,
-          "MRR": 0.5660896594773488,
-          "nDCG@k": 0.6022495608262091
+          "MRR": 0.5622940186865697,
+          "nDCG@k": 0.5966760830419942
         },
         "BM25 full-content": {
-          "recall@k": 0.8626373626373627,
+          "recall@k": 0.8516483516483516,
           "hit@1": 0.4725274725274725,
-          "MRR": 0.607356395043468,
-          "nDCG@k": 0.6624090789490626
+          "MRR": 0.6047880662244283,
+          "nDCG@k": 0.6574937765278362
         }
       }
     },
@@ -25,14 +25,14 @@
         "catalog.search": {
           "recall@k": 0.08,
           "hit@1": 0.02,
-          "MRR": 0.04203489933881709,
-          "nDCG@k": 0.04333333333333333
+          "MRR": 0.040721853968010216,
+          "nDCG@k": 0.04244796319302442
         },
         "BM25 full-content": {
-          "recall@k": 0.37,
+          "recall@k": 0.35,
           "hit@1": 0.16,
-          "MRR": 0.24542437429912048,
-          "nDCG@k": 0.26491049774398634
+          "MRR": 0.2447192741990893,
+          "nDCG@k": 0.2591292012176286
         }
       }
     }

{boost_skill_cli-1.0.358 → boost_skill_cli-1.0.359}/tests/unit/test_eval_grading.py

@@ -8,9 +8,22 @@ different skills ranked first, which makes every published number an upper
 bound.
 
 These tests pin the fix (an optional per-row `exemplar` that grades by content
-class, so mirrors of one skill still count and homonyms do not) *and* the
-property that matters more: with no exemplar, grading is byte-for-byte what it
-was, so the published baselines stay comparable.
+class, so mirrors of one skill still count and homonyms do not) and the property
+that a row without an exemplar still grades *relevance* by name, so the golden
+sets can migrate a row at a time.
+
+They also pin the second half, which the first pass got wrong. A grade key does
+two jobs: it decides whether an entry is relevant, and it is the identity the
+ranked list is de-duplicated on. Keying both on the name meant 13 genuinely
+different skills called `code-reviewer` collapsed into ONE rank slot — the eval
+crediting a ranker for a compression that exists only in the scoring code, and
+over the pinned corpus that inflated recall@10 by about one query (0.863 vs
+0.852). Exemplar rows had the mirror-image bug: their distractors keyed on
+`tap::skill_md`, so byte-identical mirrors of a distractor each took a slot.
+
+Relevance is therefore decided by name (or class), while identity is always the
+entry's content hash — one convention for every row, which is what lets an
+exemplar-graded row and a name-graded row be averaged into one number.
 """
 from __future__ import annotations
 
@@ -119,6 +132,52 @@ class TestExemplarsFailLoudly:
                            "exemplar": "no-separator"}, HASHES)
 
 
+class TestIdentityIsTheBodyNotTheName:
+    """One de-duplication convention, whether or not the row pins an exemplar."""
+
+    def test_two_different_skills_sharing_a_name_take_two_slots(self):
+        # The inflation: owner/a and owner/b are different `code-reviewer`s. A
+        # user scrolling results sees two entries, so the eval must too.
+        m = _load()
+        row = m.prepare_row({"query": "q", "relevant": ["something-else"]}, HASHES)
+        assert len(m.dedupe_keys([m.grade_key(row, A, HASHES),
+                                  m.grade_key(row, B, HASHES)])) == 2
+
+    def test_a_byte_identical_mirror_takes_one_slot(self):
+        # owner/c mirrors owner/a. Counting it twice would punish nothing and
+        # reward nothing — it is the same skill arriving from two registries.
+        m = _load()
+        row = m.prepare_row({"query": "q", "relevant": ["something-else"]}, HASHES)
+        assert len(m.dedupe_keys([m.grade_key(row, A, HASHES),
+                                  m.grade_key(row, C, HASHES)])) == 1
+
+    def test_distractor_mirrors_collapse_for_an_exemplar_row_too(self):
+        # The mirror-image bug: keyed on tap::skill_md, two mirrors of a
+        # DISTRACTOR each took a rank slot, pushing the target later.
+        m = _load()
+        row = m.prepare_row(
+            {"query": "q", "relevant": ["code-reviewer"],
+             "exemplar": "owner/b::code-reviewer/SKILL.md"}, HASHES)
+        assert len(m.dedupe_keys([m.grade_key(row, A, HASHES),
+                                  m.grade_key(row, C, HASHES)])) == 1
+
+    def test_a_relevant_entry_is_still_keyed_by_its_name(self):
+        # Relevance semantics are unchanged: recall over a multi-name `relevant`
+        # list still needs each distinct name found.
+        m = _load()
+        row = m.prepare_row({"query": "q", "relevant": ["code-reviewer"]}, HASHES)
+        assert m.grade_key(row, A, HASHES) == "code-reviewer"
+
+    def test_an_entry_with_no_content_hash_still_gets_a_unique_key(self):
+        # Degrade, don't collide: an unhashed entry must not merge with another
+        # unhashed one and silently shorten the ranked list.
+        m = _load()
+        row = m.prepare_row({"query": "q", "relevant": ["x"]}, HASHES)
+        one = _entry("p", "owner/z", "p/SKILL.md")
+        two = _entry("q", "owner/z", "q/SKILL.md")
+        assert m.grade_key(row, one, HASHES) != m.grade_key(row, two, HASHES)
+
+
 class TestDedupeKeepsTheBestRank:
     def test_repeats_collapse_to_first_occurrence(self):
         m = _load()