boost-skill-cli 1.0.353__tar.gz → 1.0.355__tar.gz

{boost_skill_cli-1.0.353 → boost_skill_cli-1.0.355}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.0.353
+Version: 1.0.355
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License: GNU GENERAL PUBLIC LICENSE

{boost_skill_cli-1.0.353 → boost_skill_cli-1.0.355}/boost_cli/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '1.0.353'
-__version_tuple__ = version_tuple = (1, 0, 353)
+__version__ = version = '1.0.355'
+__version_tuple__ = version_tuple = (1, 0, 355)
 
 __commit_id__ = commit_id = None

{boost_skill_cli-1.0.353 → boost_skill_cli-1.0.355}/boost_skill_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.0.353
+Version: 1.0.355
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License: GNU GENERAL PUBLIC LICENSE

{boost_skill_cli-1.0.353 → boost_skill_cli-1.0.355}/docs/roadmap/items/keyless-dense-tier-local-static-embeddings.md

@@ -2,12 +2,12 @@
 id: keyless-dense-tier-local-static-embeddings
 board: code
 section: pipeline
-status: planned
+status: declined
 category: Search · Retrieval
 complexity: L
 impact: High
 wow: 5
-note: spike done: the perf claims were 6-22x conservative, but rerank added +0 at real scale
+note: two models, three representations — no measurable gain over BM25, and fusion only ties
 order: 73
 owner:
 pr:
@@ -122,3 +122,38 @@ all present and rank 7th, 8th, 38th, 163rd rather than 1st. The eval corpus is n
 a real install, and the gate's floors describe a catalogue two orders of magnitude smaller than the
 one users have. Tracked separately in [[eval-corpus-is-96x-smaller-than-a-real-install]].
 
+<b>Declined on measurement, after a second model was tried specifically to avoid declining on one
+data point.</b> The card's premise is that a local static model buys keyless <em>semantic</em>
+search. Tested against the 50 natural-language golden queries over the pinned 20-tap corpus (3,843
+entries as those registries stand today), with BM25 at <code>hit@1</code> <b>0.260</b> as the
+baseline in every run:
+
+<code>potion-retrieval-32M</code> (retrieval-tuned, 63,091&times;512 F32) scored dense <b>0.220</b>,
+hybrid RRF <b>0.260</b>. <code>potion-code-16M-v2</code> (code-domain, 63,457&times;256 F16 &mdash;
+chosen because this catalogue <em>is</em> coding-agent skills, which is the strongest hypothesis for
+why a general retrieval model would underperform here) scored dense <b>0.240</b>, hybrid
+<b>0.260</b>. A third representation &mdash; name+description only, over the full 71,655-entry
+catalogue &mdash; reranked BM25's top-200 to <b>+0 net queries</b>.
+
+<b>So: two models, three representations, no measurable gain, and fusion never beats BM25 alone.</b>
+The code model is one query better than the retrieval model, which at n=50 is inside the noise
+(&plusmn;0.02 per query) and should not be read as a trend.
+
+<b>The contrast is what makes this a decline rather than a shrug.</b> The published eval measures a
+<em>real</em> embedding model at hybrid <b>0.440</b> against BM25 <b>0.340</b> &mdash; a genuine
++0.100. Static embeddings reproduce the <em>cost profile</em> that made the keyless tier attractive
+(0.16&nbsp;ms/query, 3.3&nbsp;ms/doc, no dependency) but not the <em>quality</em> that made it worth
+having. Cheap and no better than what ships today is not a tier; it is a second code path to
+maintain for nothing.
+
+<b>What survives.</b> The performance findings stand on their own and are already recorded above:
+doc-side embedding is ~24&times; faster than the card assumed, which is why prebuilt shards are an
+optimisation rather than a requirement for the <em>real</em> models in
+[[keyless-semantic-search-for-everyone]]. The pure-stdlib loader (WordPiece &rarr; gather &rarr;
+mean-pool &rarr; L2, mmap'd, F32 and F16) is proven workable if a future model justifies it.
+
+<b>What would reopen this.</b> A static model that actually separates on this task &mdash; the bar
+is beating <code>0.260</code> as a reranker, not merely producing plausible cosines. The two hand
+pairs looked fine for both models (0.224 related vs -0.001 unrelated for the code model), which is
+exactly why plausible similarity was not accepted as evidence.
+

{boost_skill_cli-1.0.353 → boost_skill_cli-1.0.355}/docs/roadmap.html

@@ -4410,7 +4410,7 @@ card no longer describes the code.</p></details>
       </article>
 
       <article class="cap rcard" id="keyless-dense-tier-local-static-embeddings">
-        <div class="head"><span class="pill planned">Planned</span><span class="cat">Search · Retrieval</span></div>
+        <div class="head"><span class="pill declined">Declined</span><span class="cat">Search · Retrieval</span></div>
         <h3>Semantic search for users who will never set an API key</h3>
         <p>Dense retrieval today needs the <code>[rag]</code> extra <em>and</em> a
 <code>VOYAGE_API_KEY</code>/<code>OPENAI_API_KEY</code> <em>and</em> a built
@@ -4519,12 +4519,47 @@ are correct: the published figure is measured over the pinned 6-tap eval corpus
 entries, and this run used a real 77-tap install &mdash; <b>96&times; larger</b>. Golden targets are
 all present and rank 7th, 8th, 38th, 163rd rather than 1st. The eval corpus is not a scale model of
 a real install, and the gate's floors describe a catalogue two orders of magnitude smaller than the
-one users have. Tracked separately in [[eval-corpus-is-96x-smaller-than-a-real-install]].</p>
+one users have. Tracked separately in [[eval-corpus-is-96x-smaller-than-a-real-install]].
+
+<b>Declined on measurement, after a second model was tried specifically to avoid declining on one
+data point.</b> The card's premise is that a local static model buys keyless <em>semantic</em>
+search. Tested against the 50 natural-language golden queries over the pinned 20-tap corpus (3,843
+entries as those registries stand today), with BM25 at <code>hit@1</code> <b>0.260</b> as the
+baseline in every run:
+
+<code>potion-retrieval-32M</code> (retrieval-tuned, 63,091&times;512 F32) scored dense <b>0.220</b>,
+hybrid RRF <b>0.260</b>. <code>potion-code-16M-v2</code> (code-domain, 63,457&times;256 F16 &mdash;
+chosen because this catalogue <em>is</em> coding-agent skills, which is the strongest hypothesis for
+why a general retrieval model would underperform here) scored dense <b>0.240</b>, hybrid
+<b>0.260</b>. A third representation &mdash; name+description only, over the full 71,655-entry
+catalogue &mdash; reranked BM25's top-200 to <b>+0 net queries</b>.
+
+<b>So: two models, three representations, no measurable gain, and fusion never beats BM25 alone.</b>
+The code model is one query better than the retrieval model, which at n=50 is inside the noise
+(&plusmn;0.02 per query) and should not be read as a trend.
+
+<b>The contrast is what makes this a decline rather than a shrug.</b> The published eval measures a
+<em>real</em> embedding model at hybrid <b>0.440</b> against BM25 <b>0.340</b> &mdash; a genuine
++0.100. Static embeddings reproduce the <em>cost profile</em> that made the keyless tier attractive
+(0.16&nbsp;ms/query, 3.3&nbsp;ms/doc, no dependency) but not the <em>quality</em> that made it worth
+having. Cheap and no better than what ships today is not a tier; it is a second code path to
+maintain for nothing.
+
+<b>What survives.</b> The performance findings stand on their own and are already recorded above:
+doc-side embedding is ~24&times; faster than the card assumed, which is why prebuilt shards are an
+optimisation rather than a requirement for the <em>real</em> models in
+[[keyless-semantic-search-for-everyone]]. The pure-stdlib loader (WordPiece &rarr; gather &rarr;
+mean-pool &rarr; L2, mmap'd, F32 and F16) is proven workable if a future model justifies it.
+
+<b>What would reopen this.</b> A static model that actually separates on this task &mdash; the bar
+is beating <code>0.260</code> as a reranker, not merely producing plausible cosines. The two hand
+pairs looked fine for both models (0.224 related vs -0.001 unrelated for the code model), which is
+exactly why plausible similarity was not accepted as evidence.</p>
         <div class="meta">
           <span class="m">Complexity <b>L</b></span>
           <span class="m hi">Impact <b>High</b></span>
           <span class="m">Wow <b class="wow">★★★★★</b></span>
-          <span class="m">spike done: the perf claims were 6-22x conservative, but rerank added +0 at real scale</span>
+          <span class="m">two models, three representations — no measurable gain over BM25, and fusion only ties</span>
         </div>
       </article>