boost-skill-cli 1.0.351__tar.gz → 1.0.352__tar.gz

{boost_skill_cli-1.0.351 → boost_skill_cli-1.0.352}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.0.351
+Version: 1.0.352
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License: GNU GENERAL PUBLIC LICENSE

{boost_skill_cli-1.0.351 → boost_skill_cli-1.0.352}/boost_cli/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '1.0.351'
-__version_tuple__ = version_tuple = (1, 0, 351)
+__version__ = version = '1.0.352'
+__version_tuple__ = version_tuple = (1, 0, 352)
 
 __commit_id__ = commit_id = None

{boost_skill_cli-1.0.351 → boost_skill_cli-1.0.352}/boost_skill_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.0.351
+Version: 1.0.352
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License: GNU GENERAL PUBLIC LICENSE

{boost_skill_cli-1.0.351 → boost_skill_cli-1.0.352}/docs/roadmap/items/eval-corpus-is-96x-smaller-than-a-real-install.md

@@ -7,7 +7,7 @@ category: Eval · Correctness
 complexity: M
 impact: High
 wow: 4
-note: BM25 hit@1 is 0.340 on the pinned corpus and 0.040 on a real one
+note: measured the falloff — hit@1 0.420 at 53 entries, 0.020 at 60k, no plateau
 order: 80
 owner:
 pr:
@@ -45,3 +45,36 @@ first step than building it.
 <b>Provenance:</b> found while running the spike in
 [[keyless-dense-tier-local-static-embeddings]], where the first sign was BM25 scoring 0.040 in a
 harness that had to be checked for a bug before the number could be believed.
+
+<b>The falloff is measured now, which was this card's own proposed first step.</b> It asked where
+between 743 and 71,655 the metrics fall off, on the grounds that knowing the shape is cheaper than
+building a second tier blind. Holding the 50 natural-language golden queries fixed and growing the
+corpus from one entry per golden name upward with random distractors:
+
+<code>53 &rarr; <b>0.420</b></code> &middot; <code>253 &rarr; 0.380</code> &middot;
+<code>753 &rarr; 0.240</code> &middot; <code>2,053 &rarr; 0.220</code> &middot;
+<code>6,053 &rarr; 0.080</code> &middot; <code>20,053 &rarr; 0.040</code> &middot;
+<code>60,053 &rarr; <b>0.020</b></code>
+
+<b>There is no plateau, and that is the answer.</b> <code>hit@1</code> decays continuously &mdash;
+roughly halving for every 4&times; the corpus grows &mdash; rather than degrading to some floor a
+larger fixed corpus would capture. So &ldquo;pick a bigger number for the second tier&rdquo; has no
+principled stopping point; what the gate can honestly claim is bounded by the size it measures, and
+that size is ~750 while users run 10&ndash;100&times; more. A scheduled tier is still worth having,
+but it should be described as <em>a</em> scale rather than <em>the</em> scale.
+
+<b>A second finding fell out of the method, and it complicates the metric itself.</b> The first
+attempt grew the corpus tap-by-tap and could not get below <b>51,657</b> entries while keeping every
+golden target present &mdash; because <b>119 separate taps</b> ship a skill matching a golden name.
+Names like <code>review</code>, <code>commit</code> and <code>why</code> are not identifying. So
+<code>hit@1</code> graded <em>by name</em> counts a hit when any of 119 same-named skills lands
+first, which flatters the metric at scale exactly where it looks worst. Grading by
+<code>(tap, skill_md)</code> would measure what a user actually needs; that is a bigger change than
+this card, and worth its own.
+
+<b>Stated limits.</b> BM25 IDF still comes from the full index rather than being recomputed per
+subset, so these are the ranking effects of added <em>candidates</em>, not a byte-exact simulation
+of a small install. Distractors are sampled uniformly from real entries rather than composed as a
+plausible tap set. And n=50, so one query is &plusmn;0.02 &mdash; the endpoints are far apart enough
+to carry the conclusion, the middle steps are not individually significant.
+

{boost_skill_cli-1.0.351 → boost_skill_cli-1.0.352}/docs/roadmap.html

@@ -2277,12 +2277,44 @@ first step than building it.
 
 <b>Provenance:</b> found while running the spike in
 [[keyless-dense-tier-local-static-embeddings]], where the first sign was BM25 scoring 0.040 in a
-harness that had to be checked for a bug before the number could be believed.</p>
+harness that had to be checked for a bug before the number could be believed.
+
+<b>The falloff is measured now, which was this card's own proposed first step.</b> It asked where
+between 743 and 71,655 the metrics fall off, on the grounds that knowing the shape is cheaper than
+building a second tier blind. Holding the 50 natural-language golden queries fixed and growing the
+corpus from one entry per golden name upward with random distractors:
+
+<code>53 &rarr; <b>0.420</b></code> &middot; <code>253 &rarr; 0.380</code> &middot;
+<code>753 &rarr; 0.240</code> &middot; <code>2,053 &rarr; 0.220</code> &middot;
+<code>6,053 &rarr; 0.080</code> &middot; <code>20,053 &rarr; 0.040</code> &middot;
+<code>60,053 &rarr; <b>0.020</b></code>
+
+<b>There is no plateau, and that is the answer.</b> <code>hit@1</code> decays continuously &mdash;
+roughly halving for every 4&times; the corpus grows &mdash; rather than degrading to some floor a
+larger fixed corpus would capture. So &ldquo;pick a bigger number for the second tier&rdquo; has no
+principled stopping point; what the gate can honestly claim is bounded by the size it measures, and
+that size is ~750 while users run 10&ndash;100&times; more. A scheduled tier is still worth having,
+but it should be described as <em>a</em> scale rather than <em>the</em> scale.
+
+<b>A second finding fell out of the method, and it complicates the metric itself.</b> The first
+attempt grew the corpus tap-by-tap and could not get below <b>51,657</b> entries while keeping every
+golden target present &mdash; because <b>119 separate taps</b> ship a skill matching a golden name.
+Names like <code>review</code>, <code>commit</code> and <code>why</code> are not identifying. So
+<code>hit@1</code> graded <em>by name</em> counts a hit when any of 119 same-named skills lands
+first, which flatters the metric at scale exactly where it looks worst. Grading by
+<code>(tap, skill_md)</code> would measure what a user actually needs; that is a bigger change than
+this card, and worth its own.
+
+<b>Stated limits.</b> BM25 IDF still comes from the full index rather than being recomputed per
+subset, so these are the ranking effects of added <em>candidates</em>, not a byte-exact simulation
+of a small install. Distractors are sampled uniformly from real entries rather than composed as a
+plausible tap set. And n=50, so one query is &plusmn;0.02 &mdash; the endpoints are far apart enough
+to carry the conclusion, the middle steps are not individually significant.</p>
         <div class="meta">
           <span class="m">Complexity <b>M</b></span>
           <span class="m hi">Impact <b>High</b></span>
           <span class="m">Wow <b class="wow">★★★★</b></span>
-          <span class="m">BM25 hit@1 is 0.340 on the pinned corpus and 0.040 on a real one</span>
+          <span class="m">measured the falloff — hit@1 0.420 at 53 entries, 0.020 at 60k, no plateau</span>
         </div>
       </article>
       <!-- /ROADMAP:cards -->