boost-skill-cli 1.0.310__tar.gz → 1.0.312__tar.gz

{boost_skill_cli-1.0.310/boost_skill_cli.egg-info → boost_skill_cli-1.0.312}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.0.310
+Version: 1.0.312
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License: GNU GENERAL PUBLIC LICENSE

{boost_skill_cli-1.0.310 → boost_skill_cli-1.0.312}/boost_cli/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '1.0.310'
-__version_tuple__ = version_tuple = (1, 0, 310)
+__version__ = version = '1.0.312'
+__version_tuple__ = version_tuple = (1, 0, 312)
 
 __commit_id__ = commit_id = None

{boost_skill_cli-1.0.310 → boost_skill_cli-1.0.312/boost_skill_cli.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.0.310
+Version: 1.0.312
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License: GNU GENERAL PUBLIC LICENSE

{boost_skill_cli-1.0.310 → boost_skill_cli-1.0.312}/docs/roadmap/items/keyless-semantic-search-for-everyone.md

@@ -130,7 +130,67 @@ not reproduce here, though import cost is genuinely machine- and version-depende
 may be real on theirs. 233&nbsp;ms is also very likely <i>faster</i> than the Voyage round trip it
 sits beside, and it is paid only by users who installed the extra.
 
+<b>If step 6 is the tie-break, it needs a method and not just a number.</b> The sibling item already
+produced one cautionary result worth inheriting rather than repeating: its own headline, <b>+11.0
+recall / +15.9 hit@1</b>, did not survive verification. Three failures, none specific to a static
+model, all reachable from here. Its baseline used a kind oracle the live search path does not have,
+so the comparison was never against what a user experiences. The blend weight
+(<code>w_dense=0.7</code>) <em>and</em> the rerank pool depth were both tuned by argmax on the same
+82 queries they were then reported on &mdash; fitting and reporting on one set. And at n=82 on a
+binary metric, the smallest net win reaching <em>p</em>&lt;0.05 is <b>6 queries</b>: its
+<code>hit@1</code> (+13 net) clears that bar, while recall (+9) sits at the resolution floor and
+should not have led.
+
+So four constraints on step 6, whichever backend wins. Report <b>McNemar</b> on paired per-query
+outcomes rather than two independent-looking averages, since the engines are scored on the same
+queries. Hold the blend weight and pool depth <b>out</b> of the query set they are scored on. Lead
+with <code>hit@1</code> &mdash; on a golden set this size, recall moves inside its own noise. And
+measure each engine <b>alone</b> before any fusion, so a hybrid win cannot be quietly credited to
+the embedder that did not earn it. One structural caveat also transfers: a skill's name is only
+~10.5% of a mean-pooled surface vector while 106 description clusters are shared across 270 distinct
+names, so a lift measured on today's corpus may <em>shrink</em> rather than hold as the index grows
+toward 50k items &mdash; an argument for re-measuring at scale before believing any of this.
+
 The static approach still wins on cost and would win outright if the quality gap is small &mdash;
 which is exactly what the eval should decide. Its own card is right that neither should ship a
 retrieval <i>claim</i> before that eval exists.
 
+<b>Step 6, run against those four constraints &mdash; and it does not say what this card assumed.</b>
+Corpus rebuilt from <code>tests/eval/taps.txt</code> (6 taps, 743 entries, 3740 passages embedded
+locally), k=10, 91 golden queries. Each engine measured <em>alone</em>, no fusion:
+<code>catalog.search</code> hit@1 0.714, recall 0.918, MRR 0.783; <b>BM25 full-content hit@1
+0.780</b>, recall 1.000, MRR 0.860; <b>dense (local bge-small) hit@1 0.780</b>, recall 0.956, MRR
+0.853.
+
+Leading with <code>hit@1</code> as instructed: <b>71 queries each &mdash; an exact tie.</b> Recall
+differs by 4 queries (91 against 87) and MRR by 0.6, both under the ~6-query floor this card sets
+for <em>p</em>&lt;0.05 at this <em>n</em>. So the honest reading is <b>no significant difference
+between BM25 and local dense on the golden set</b> &mdash; not a win for either. An earlier draft of
+this note claimed BM25 won; that was the recall number leading, which is exactly the error the four
+constraints above were written to prevent.
+
+<b>The more useful result is that the golden set cannot grade this feature at all.</b> It scores
+real catalog items <em>by name</em>, which is BM25's strength by construction &mdash;
+<code>CLAUDE.md</code> already records that BM25 recall over this corpus is 1.000. It contains none
+of the human-phrased queries the keyless work exists for, and on those the two engines separate
+sharply: <code>"my app is slow"</code> returns <code>phoenix-docker-setup</code>,
+<code>guidelines</code>, <code>solidjs---error-boundaries</code> from BM25 against
+<code>analyse-problem</code>, <b><code>performance-optimization</code></b>,
+<code>fastapi-best-practices</code> from dense; <code>"I need to make my website accessible"</code>
+returns <code>do-and-judge</code>, <code>write-concisely</code>, <code>do-in-steps</code> against
+<b><code>accessibility-guidelines</code></b> first. That is a demonstration, not a measurement
+&mdash; there is no scored query set of that shape yet, which is the point.
+
+Three consequences. <b>Step 6's first deliverable is a query set, not a number</b>: golden queries
+in the human-phrased style, or the eval keeps answering a question nobody asked. <b>Step 4 (RRF)
+gains support</b> &mdash; a tie on keyword queries plus a qualitative dense win on human ones is the
+"two engines fail in opposite directions" case, and fusing is the response to it. And <b>the current
+preference order deserves a look</b>: <code>rag.retrieve_any</code> takes dense whenever it is ready
+and non-empty, so shipping step 1 without step 4 silently moved keyword queries onto the engine that
+is, at best, tied for them.
+
+<b>Method note.</b> The first indexing pass reported 3 taps failed to embed (2716 of 3740 passages).
+A rerun stored all 3740, and two 30-batch replays never reproduced a failure, so it was transient
+resource pressure rather than a defect &mdash; the retry path (no commit recorded for a failed tap)
+did its job. The figures above come from a complete index.
+

{boost_skill_cli-1.0.310 → boost_skill_cli-1.0.312}/docs/roadmap.html

@@ -718,9 +718,69 @@ not reproduce here, though import cost is genuinely machine- and version-depende
 may be real on theirs. 233&nbsp;ms is also very likely <i>faster</i> than the Voyage round trip it
 sits beside, and it is paid only by users who installed the extra.
 
+<b>If step 6 is the tie-break, it needs a method and not just a number.</b> The sibling item already
+produced one cautionary result worth inheriting rather than repeating: its own headline, <b>+11.0
+recall / +15.9 hit@1</b>, did not survive verification. Three failures, none specific to a static
+model, all reachable from here. Its baseline used a kind oracle the live search path does not have,
+so the comparison was never against what a user experiences. The blend weight
+(<code>w_dense=0.7</code>) <em>and</em> the rerank pool depth were both tuned by argmax on the same
+82 queries they were then reported on &mdash; fitting and reporting on one set. And at n=82 on a
+binary metric, the smallest net win reaching <em>p</em>&lt;0.05 is <b>6 queries</b>: its
+<code>hit@1</code> (+13 net) clears that bar, while recall (+9) sits at the resolution floor and
+should not have led.
+
+So four constraints on step 6, whichever backend wins. Report <b>McNemar</b> on paired per-query
+outcomes rather than two independent-looking averages, since the engines are scored on the same
+queries. Hold the blend weight and pool depth <b>out</b> of the query set they are scored on. Lead
+with <code>hit@1</code> &mdash; on a golden set this size, recall moves inside its own noise. And
+measure each engine <b>alone</b> before any fusion, so a hybrid win cannot be quietly credited to
+the embedder that did not earn it. One structural caveat also transfers: a skill's name is only
+~10.5% of a mean-pooled surface vector while 106 description clusters are shared across 270 distinct
+names, so a lift measured on today's corpus may <em>shrink</em> rather than hold as the index grows
+toward 50k items &mdash; an argument for re-measuring at scale before believing any of this.
+
 The static approach still wins on cost and would win outright if the quality gap is small &mdash;
 which is exactly what the eval should decide. Its own card is right that neither should ship a
-retrieval <i>claim</i> before that eval exists.</p>
+retrieval <i>claim</i> before that eval exists.
+
+<b>Step 6, run against those four constraints &mdash; and it does not say what this card assumed.</b>
+Corpus rebuilt from <code>tests/eval/taps.txt</code> (6 taps, 743 entries, 3740 passages embedded
+locally), k=10, 91 golden queries. Each engine measured <em>alone</em>, no fusion:
+<code>catalog.search</code> hit@1 0.714, recall 0.918, MRR 0.783; <b>BM25 full-content hit@1
+0.780</b>, recall 1.000, MRR 0.860; <b>dense (local bge-small) hit@1 0.780</b>, recall 0.956, MRR
+0.853.
+
+Leading with <code>hit@1</code> as instructed: <b>71 queries each &mdash; an exact tie.</b> Recall
+differs by 4 queries (91 against 87) and MRR by 0.6, both under the ~6-query floor this card sets
+for <em>p</em>&lt;0.05 at this <em>n</em>. So the honest reading is <b>no significant difference
+between BM25 and local dense on the golden set</b> &mdash; not a win for either. An earlier draft of
+this note claimed BM25 won; that was the recall number leading, which is exactly the error the four
+constraints above were written to prevent.
+
+<b>The more useful result is that the golden set cannot grade this feature at all.</b> It scores
+real catalog items <em>by name</em>, which is BM25's strength by construction &mdash;
+<code>CLAUDE.md</code> already records that BM25 recall over this corpus is 1.000. It contains none
+of the human-phrased queries the keyless work exists for, and on those the two engines separate
+sharply: <code>"my app is slow"</code> returns <code>phoenix-docker-setup</code>,
+<code>guidelines</code>, <code>solidjs---error-boundaries</code> from BM25 against
+<code>analyse-problem</code>, <b><code>performance-optimization</code></b>,
+<code>fastapi-best-practices</code> from dense; <code>"I need to make my website accessible"</code>
+returns <code>do-and-judge</code>, <code>write-concisely</code>, <code>do-in-steps</code> against
+<b><code>accessibility-guidelines</code></b> first. That is a demonstration, not a measurement
+&mdash; there is no scored query set of that shape yet, which is the point.
+
+Three consequences. <b>Step 6's first deliverable is a query set, not a number</b>: golden queries
+in the human-phrased style, or the eval keeps answering a question nobody asked. <b>Step 4 (RRF)
+gains support</b> &mdash; a tie on keyword queries plus a qualitative dense win on human ones is the
+"two engines fail in opposite directions" case, and fusing is the response to it. And <b>the current
+preference order deserves a look</b>: <code>rag.retrieve_any</code> takes dense whenever it is ready
+and non-empty, so shipping step 1 without step 4 silently moved keyword queries onto the engine that
+is, at best, tied for them.
+
+<b>Method note.</b> The first indexing pass reported 3 taps failed to embed (2716 of 3740 passages).
+A rerun stored all 3740, and two 30-batch replays never reproduced a failure, so it was transient
+resource pressure rather than a defect &mdash; the retry path (no commit recorded for a failed tap)
+did its job. The figures above come from a complete index.</p>
         <div class="meta">
           <span class="m">Complexity <b>L</b></span>
           <span class="m hi">Impact <b>High</b></span>