boost-skill-cli 1.0.352__tar.gz → 1.0.353__tar.gz

{boost_skill_cli-1.0.352 → boost_skill_cli-1.0.353}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.0.352
+Version: 1.0.353
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License: GNU GENERAL PUBLIC LICENSE

{boost_skill_cli-1.0.352 → boost_skill_cli-1.0.353}/boost_cli/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '1.0.352'
-__version_tuple__ = version_tuple = (1, 0, 352)
+__version__ = version = '1.0.353'
+__version_tuple__ = version_tuple = (1, 0, 353)
 
 __commit_id__ = commit_id = None

{boost_skill_cli-1.0.352 → boost_skill_cli-1.0.353}/boost_skill_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.0.352
+Version: 1.0.353
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License: GNU GENERAL PUBLIC LICENSE

{boost_skill_cli-1.0.352 → boost_skill_cli-1.0.353}/boost_skill_cli.egg-info/SOURCES.txt

@@ -225,6 +225,7 @@ docs/roadmap/items/frontmatter-scalar-over-coercion.md
 docs/roadmap/items/gemini-cli-agent-target.md
 docs/roadmap/items/github-community-health-files.md
 docs/roadmap/items/github-pages-deploy-is-broken-every-push.md
+docs/roadmap/items/golden-set-grades-by-name-not-by-skill.md
 docs/roadmap/items/golden-set-statistical-power.md
 docs/roadmap/items/hash-pinned-reproducible-toolchain-uv-lock.md
 docs/roadmap/items/html-validation-html-validate.md

boost_skill_cli-1.0.353/docs/roadmap/items/golden-set-grades-by-name-not-by-skill.md

@@ -0,0 +1,57 @@
+---
+id: golden-set-grades-by-name-not-by-skill
+board: code
+section: internals
+status: planned
+category: Eval · Correctness
+complexity: M
+impact: High
+wow: 4
+note: 35 of 53 golden targets are homonyms — code-reviewer is 79 copies across 59 different skills
+order: 81
+owner:
+pr:
+title: The golden set grades by name, and 35 of 53 names are ambiguous
+---
+<b>The eval scores a hit when the top result carries the right <em>name</em>. Most of those names do
+not identify a skill.</b> Measured over a real 71,655-entry catalogue against the 50
+natural-language golden queries: of the <b>53</b> distinct target names, <b>18</b> resolve to a
+single body &mdash; harmless mirrors of one skill across registries &mdash; and <b>35</b> resolve to
+<em>more than one</em>, which are genuinely different skills that happen to share a name.
+
+The worst are not marginal. <code>code-reviewer</code> exists as <b>79 copies across 59 distinct
+contents</b>. <code>skill-creator</code> is 77 copies / 30 contents, <code>frontend-design</code>
+81 / 18, <code>commit</code> 40 / 25. In total <b>822 entries match a golden name, across 351
+distinct bodies</b> &mdash; so roughly one in seven of the entries that can satisfy a golden query
+is the one the query was written about.
+
+<b>Why this is a correctness problem and not a rounding error.</b> A query graded against
+<code>code-reviewer</code> scores a hit when any one of 59 different skills ranks first, including
+ones that review a different language, target a different agent, or do something else entirely. The
+metric therefore reports an <b>upper bound</b> on retrieval quality, and it is loosest exactly where
+the catalogue is largest &mdash; the regime
+[[eval-corpus-is-96x-smaller-than-a-real-install]] shows the gate never measures at all. Every
+decision validated against it (blend weights, pool depth, whether a reranker earns its keep)
+inherits that looseness.
+
+<b>The fix is not mechanical, which is why this is a card and not a patch.</b> Grading by
+<code>(tap, skill_md)</code> would measure what a user actually needs, and that key is already known
+to be unique &mdash; <code>#366</code> moved the catalogue onto it for exactly this reason. But
+retargeting the golden set means deciding <em>which</em> of 59 <code>code-reviewer</code>s a query
+about reviewing a diff for security problems should be graded against. That is a judgment about
+intent, not a lookup, and guessing it would quietly bake one opinion into the number the project
+reports. Two defensible shapes: pin each golden row to a specific <code>tap</code> +
+<code>skill_md</code>, or accept any entry whose body falls in a named <b>equivalence class</b>, so
+mirrors still count and homonyms do not.
+
+<b>What this does not claim.</b> It does not follow that retrieval is worse than reported in
+proportion. A same-named alternative is often a perfectly reasonable answer, which is why the
+eval's <code>relevant</code> field is a list rather than a single value. What is established is
+that the number cannot <em>distinguish</em> the two cases, so it must not be read as precision
+about the intended skill.
+
+<b>Provenance.</b> Surfaced while measuring the scale falloff, where growing the corpus tap-by-tap
+could not get below 51,657 entries without dropping a golden target &mdash; <b>119</b> taps ship a
+skill matching one of these names. It also qualifies an earlier claim of mine: &ldquo;all 50 targets
+are present&rdquo;, reported while diagnosing the static-embedding spike, was true but matched by
+name, so it was weaker evidence than it read as.

{boost_skill_cli-1.0.352 → boost_skill_cli-1.0.353}/docs/roadmap.html

@@ -125,7 +125,7 @@
       <div class="stat"><b>16</b><span>Shipped</span></div>
       <div class="stat"><b>1</b><span>Next up</span></div>
       <div class="stat"><b>4</b><span>Planned</span></div>
-      <div class="stat"><b>195</b><span>Loop finds</span></div>
+      <div class="stat"><b>196</b><span>Loop finds</span></div>
       <!-- /ROADMAP:stats -->
     </div>
   </section>
@@ -2317,6 +2317,59 @@ to carry the conclusion, the middle steps are not individually significant.</p>
           <span class="m">measured the falloff — hit@1 0.420 at 53 entries, 0.020 at 60k, no plateau</span>
         </div>
       </article>
+
+      <article class="cap rcard" id="golden-set-grades-by-name-not-by-skill">
+        <div class="head"><span class="pill planned">Planned</span><span class="cat">Eval · Correctness</span></div>
+        <h3>The golden set grades by name, and 35 of 53 names are ambiguous</h3>
+        <p><b>The eval scores a hit when the top result carries the right <em>name</em>. Most of those names do
+not identify a skill.</b> Measured over a real 71,655-entry catalogue against the 50
+natural-language golden queries: of the <b>53</b> distinct target names, <b>18</b> resolve to a
+single body &mdash; harmless mirrors of one skill across registries &mdash; and <b>35</b> resolve to
+<em>more than one</em>, which are genuinely different skills that happen to share a name.
+
+The worst are not marginal. <code>code-reviewer</code> exists as <b>79 copies across 59 distinct
+contents</b>. <code>skill-creator</code> is 77 copies / 30 contents, <code>frontend-design</code>
+81 / 18, <code>commit</code> 40 / 25. In total <b>822 entries match a golden name, across 351
+distinct bodies</b> &mdash; so roughly one in seven of the entries that can satisfy a golden query
+is the one the query was written about.
+
+<b>Why this is a correctness problem and not a rounding error.</b> A query graded against
+<code>code-reviewer</code> scores a hit when any one of 59 different skills ranks first, including
+ones that review a different language, target a different agent, or do something else entirely. The
+metric therefore reports an <b>upper bound</b> on retrieval quality, and it is loosest exactly where
+the catalogue is largest &mdash; the regime
+[[eval-corpus-is-96x-smaller-than-a-real-install]] shows the gate never measures at all. Every
+decision validated against it (blend weights, pool depth, whether a reranker earns its keep)
+inherits that looseness.
+
+<b>The fix is not mechanical, which is why this is a card and not a patch.</b> Grading by
+<code>(tap, skill_md)</code> would measure what a user actually needs, and that key is already known
+to be unique &mdash; <code>#366</code> moved the catalogue onto it for exactly this reason. But
+retargeting the golden set means deciding <em>which</em> of 59 <code>code-reviewer</code>s a query
+about reviewing a diff for security problems should be graded against. That is a judgment about
+intent, not a lookup, and guessing it would quietly bake one opinion into the number the project
+reports. Two defensible shapes: pin each golden row to a specific <code>tap</code> +
+<code>skill_md</code>, or accept any entry whose body falls in a named <b>equivalence class</b>, so
+mirrors still count and homonyms do not.
+
+<b>What this does not claim.</b> It does not follow that retrieval is worse than reported in
+proportion. A same-named alternative is often a perfectly reasonable answer, which is why the
+eval's <code>relevant</code> field is a list rather than a single value. What is established is
+that the number cannot <em>distinguish</em> the two cases, so it must not be read as precision
+about the intended skill.
+
+<b>Provenance.</b> Surfaced while measuring the scale falloff, where growing the corpus tap-by-tap
+could not get below 51,657 entries without dropping a golden target &mdash; <b>119</b> taps ship a
+skill matching one of these names. It also qualifies an earlier claim of mine: &ldquo;all 50 targets
+are present&rdquo;, reported while diagnosing the static-embedding spike, was true but matched by
+name, so it was weaker evidence than it read as.</p>
+        <div class="meta">
+          <span class="m">Complexity <b>M</b></span>
+          <span class="m hi">Impact <b>High</b></span>
+          <span class="m">Wow <b class="wow">★★★★</b></span>
+          <span class="m">35 of 53 golden targets are homonyms — code-reviewer is 79 copies across 59 different skills</span>
+        </div>
+      </article>
       <!-- /ROADMAP:cards -->
 
     </div>