boost-skill-cli 1.0.349__tar.gz → 1.0.350__tar.gz

{boost_skill_cli-1.0.349 → boost_skill_cli-1.0.350}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.0.349
+Version: 1.0.350
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License: GNU GENERAL PUBLIC LICENSE

{boost_skill_cli-1.0.349 → boost_skill_cli-1.0.350}/boost_cli/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '1.0.349'
-__version_tuple__ = version_tuple = (1, 0, 349)
+__version__ = version = '1.0.350'
+__version_tuple__ = version_tuple = (1, 0, 350)
 
 __commit_id__ = commit_id = None

{boost_skill_cli-1.0.349 → boost_skill_cli-1.0.350}/boost_skill_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.0.349
+Version: 1.0.350
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License: GNU GENERAL PUBLIC LICENSE

{boost_skill_cli-1.0.349 → boost_skill_cli-1.0.350}/boost_skill_cli.egg-info/SOURCES.txt

@@ -211,6 +211,7 @@ docs/roadmap/items/docstring-coverage-interrogate.md
 docs/roadmap/items/doctor-checks-rules-and-workflows.md
 docs/roadmap/items/end-of-options-guard-on-git-commands.md
 docs/roadmap/items/eval-corpus-cannot-see-real-retrieval.md
+docs/roadmap/items/eval-corpus-is-96x-smaller-than-a-real-install.md
 docs/roadmap/items/extension-free-tests-core-dense-py.md
 docs/roadmap/items/extract-mcp-http-servers-out-of-configuration-py.md
 docs/roadmap/items/finish-mutation-hardening-across-core.md

boost_skill_cli-1.0.350/docs/roadmap/items/eval-corpus-is-96x-smaller-than-a-real-install.md

@@ -0,0 +1,47 @@
+---
+id: eval-corpus-is-96x-smaller-than-a-real-install
+board: code
+section: internals
+status: planned
+category: Eval · Correctness
+complexity: M
+impact: High
+wow: 4
+note: BM25 hit@1 is 0.340 on the pinned corpus and 0.040 on a real one
+order: 80
+owner:
+pr:
+title: The eval corpus is 96× smaller than a real install, and the floors describe the small one
+---
+<b>The required <code>eval</code> gate measures a catalogue almost nobody has.</b>
+<code>tests/eval/taps.txt</code> pins 20 registries resolving to <b>743</b> entries. A real install
+that has run <code>boost tap</code> a few times is far bigger &mdash; the machine this was measured
+on carries <b>71,655</b> entries across 77 taps, <b>96&times;</b> larger.
+
+<b>The gap changes the answer, not just the margin.</b> Over the same 50 natural-language golden
+queries, BM25 scores <code>hit@1</code> <b>0.340</b> on the pinned corpus (published in
+<code>#373</code>) and <b>0.040</b> on the 71,655-entry one. Both numbers are correct; they are
+measurements of different things. The targets have not gone missing &mdash; all 50 are present, and
+spot-checking puts them at rank <b>7</b>, <b>8</b>, <b>38</b> and <b>163</b> rather than rank 1.
+What changed is the number of plausible distractors competing for the top slot.
+
+<b>Why this matters more than a number moving.</b> The <code>eval</code> gate floors BM25 recall@k
+at 0.85 and passes with wide margin, which reads as &ldquo;retrieval is healthy&rdquo;. It is
+healthy <em>at 743 entries</em>. Every retrieval decision validated against that corpus &mdash;
+blend weights, pool depths, whether a reranker earns its keep &mdash; is being validated at a scale
+users leave behind after their third tap. The near-duplicate card predicted exactly this shape when
+it warned the lift may <em>shrink</em> toward 50k rather than hold; this is that warning arriving
+with a number attached.
+
+<b>What is not being proposed.</b> Not lowering the floor, and not making the corpus enormous:
+<code>ensure_eval_corpus.sh</code> already taps 20 repos over the network inside the
+<code>lint</code> job, and a 77-tap corpus would make the required gate slow and flaky. The useful
+shape is probably a <b>second, larger tier</b> &mdash; scheduled rather than required, floored on
+its own baseline &mdash; so the small corpus keeps gating every PR cheaply while the large one
+catches the scale effects the small one structurally cannot. Measuring <em>where</em> between 743
+and 71,655 the metrics fall off would tell you how large that tier needs to be, and is a smaller
+first step than building it.
+
+<b>Provenance:</b> found while running the spike in
+[[keyless-dense-tier-local-static-embeddings]], where the first sign was BM25 scoring 0.040 in a
+harness that had to be checked for a bug before the number could be believed.

{boost_skill_cli-1.0.349 → boost_skill_cli-1.0.350}/docs/roadmap/items/keyless-dense-tier-local-static-embeddings.md

@@ -7,7 +7,7 @@ category: Search · Retrieval
 complexity: L
 impact: High
 wow: 5
-note: semantic search with no key, no billing, no deps
+note: spike done: the perf claims were 6-22x conservative, but rerank added +0 at real scale
 order: 73
 owner:
 pr:
@@ -77,3 +77,48 @@ that holds it is a difference of more than an order of magnitude on the doc side
 the cost that makes prebuilt shards mandatory today. Worth measuring the model2vec path directly
 before committing — but the gap it claims to close is now a measured number rather than an
 estimate.
+
+<b>Spike done &mdash; the prerequisites this card set have all shipped, so the measurement it asked
+for was finally runnable.</b> It says &ldquo;do not ship before the eval and dedup items&rdquo;;
+dedup landed in <code>#370</code>, the index format in <code>#367</code>/<code>#371</code>, the
+published eval in <code>#373</code>. What follows is <code>potion-retrieval-32M</code> (MIT, 63,091
+&times; 512 F32 lookup table &mdash; confirmed a single tensor, no transformer) driven by a
+hand-written pure-stdlib loader: WordPiece &rarr; gather rows &rarr; mean-pool &rarr; L2, mmap'd,
+no numpy.
+
+<b>The two unverified performance claims were not just right, they were conservative.</b> Query
+embedding measured <b>0.16&nbsp;ms</b> against the card's <code>~1&nbsp;ms</code>. Document
+embedding measured <b>1.34&nbsp;ms</b> on a synthetic 105-token doc and <b>3.27&nbsp;ms</b> on 300
+real catalogue entries (median 61 tokens), against the card's <code>~29&nbsp;ms</code>.
+
+<b>That reverses one of this card's design arguments.</b> The doc-side cost was the reason prebuilt
+artifacts looked mandatory: &ldquo;29&nbsp;ms/doc in pure Python is ~24&nbsp;min for 50k
+single-core&rdquo;. At the measured 3.27&nbsp;ms it is <b>2.7&nbsp;min</b> &mdash; roughly the time
+a first <code>boost tap --defaults</code> already takes. Local embedding is therefore viable on the
+user's own machine, and shipped shards become a genuine optimisation rather than a requirement. (Not
+to be confused with the ONNX <code>bge-small</code> path measured at ~1.2&nbsp;s/chunk in
+<code>keyless-semantic-search-for-everyone</code>; that number stands, and the gap between them
+<em>is</em> the case for the static model.)
+
+<b>But reranking bought nothing at real scale, which is the result that matters.</b> Over the 50
+natural-language golden queries against a real <b>71,655-entry</b> catalogue, reranking BM25's
+top-200 by cosine scored <code>hit@1</code> <b>2/50</b> &mdash; identical to BM25's own
+<b>2/50</b>, a net change of <b>+0 queries</b> where this card's own statistics note says 6 net
+queries is the smallest win reaching p&lt;0.05. On two hand-checked pairs the ordering was right but
+the margin was thin (related 0.154 vs unrelated 0.097).
+
+<b>Stated limits, because this does not settle the question.</b> The document vector was built from
+<code>name + description</code> truncated to 1,500 characters, not the full body the real dense path
+indexes, so this measures a weaker representation than the one being proposed. No blend was tried
+&mdash; pure rerank, no <code>w_dense</code> &mdash; and this card explicitly asks for a held-out
+blend weight and McNemar. What it does establish is that the <em>cheap</em> version of the idea does
+not pay for itself, so the remaining work is representation and blending, not inference speed.
+
+<b>An unrelated finding fell out of it, and it is the more important one.</b> BM25 scored
+<code>hit@1</code> <b>0.040</b> here against the <b>0.340</b> published in <code>#373</code>. Both
+are correct: the published figure is measured over the pinned 6-tap eval corpus of <b>743</b>
+entries, and this run used a real 77-tap install &mdash; <b>96&times; larger</b>. Golden targets are
+all present and rank 7th, 8th, 38th, 163rd rather than 1st. The eval corpus is not a scale model of
+a real install, and the gate's floors describe a catalogue two orders of magnitude smaller than the
+one users have. Tracked separately in [[eval-corpus-is-96x-smaller-than-a-real-install]].
+

{boost_skill_cli-1.0.349 → boost_skill_cli-1.0.350}/docs/roadmap.html

@@ -125,7 +125,7 @@
       <div class="stat"><b>16</b><span>Shipped</span></div>
       <div class="stat"><b>1</b><span>Next up</span></div>
       <div class="stat"><b>4</b><span>Planned</span></div>
-      <div class="stat"><b>194</b><span>Loop finds</span></div>
+      <div class="stat"><b>195</b><span>Loop finds</span></div>
       <!-- /ROADMAP:stats -->
     </div>
   </section>
@@ -2242,6 +2242,49 @@ killed by measurement rather than argument, and the one that cracked it was noti
           <span class="m">reds an unrelated PR, passes on rerun, cause unproven</span>
         </div>
       </article>
+
+      <article class="cap rcard" id="eval-corpus-is-96x-smaller-than-a-real-install">
+        <div class="head"><span class="pill planned">Planned</span><span class="cat">Eval · Correctness</span></div>
+        <h3>The eval corpus is 96&times; smaller than a real install, and the floors describe the small one</h3>
+        <p><b>The required <code>eval</code> gate measures a catalogue almost nobody has.</b>
+<code>tests/eval/taps.txt</code> pins 20 registries resolving to <b>743</b> entries. A real install
+that has run <code>boost tap</code> a few times is far bigger &mdash; the machine this was measured
+on carries <b>71,655</b> entries across 77 taps, <b>96&times;</b> larger.
+
+<b>The gap changes the answer, not just the margin.</b> Over the same 50 natural-language golden
+queries, BM25 scores <code>hit@1</code> <b>0.340</b> on the pinned corpus (published in
+<code>#373</code>) and <b>0.040</b> on the 71,655-entry one. Both numbers are correct; they are
+measurements of different things. The targets have not gone missing &mdash; all 50 are present, and
+spot-checking puts them at rank <b>7</b>, <b>8</b>, <b>38</b> and <b>163</b> rather than rank 1.
+What changed is the number of plausible distractors competing for the top slot.
+
+<b>Why this matters more than a number moving.</b> The <code>eval</code> gate floors BM25 recall@k
+at 0.85 and passes with wide margin, which reads as &ldquo;retrieval is healthy&rdquo;. It is
+healthy <em>at 743 entries</em>. Every retrieval decision validated against that corpus &mdash;
+blend weights, pool depths, whether a reranker earns its keep &mdash; is being validated at a scale
+users leave behind after their third tap. The near-duplicate card predicted exactly this shape when
+it warned the lift may <em>shrink</em> toward 50k rather than hold; this is that warning arriving
+with a number attached.
+
+<b>What is not being proposed.</b> Not lowering the floor, and not making the corpus enormous:
+<code>ensure_eval_corpus.sh</code> already taps 20 repos over the network inside the
+<code>lint</code> job, and a 77-tap corpus would make the required gate slow and flaky. The useful
+shape is probably a <b>second, larger tier</b> &mdash; scheduled rather than required, floored on
+its own baseline &mdash; so the small corpus keeps gating every PR cheaply while the large one
+catches the scale effects the small one structurally cannot. Measuring <em>where</em> between 743
+and 71,655 the metrics fall off would tell you how large that tier needs to be, and is a smaller
+first step than building it.
+
+<b>Provenance:</b> found while running the spike in
+[[keyless-dense-tier-local-static-embeddings]], where the first sign was BM25 scoring 0.040 in a
+harness that had to be checked for a bug before the number could be believed.</p>
+        <div class="meta">
+          <span class="m">Complexity <b>M</b></span>
+          <span class="m hi">Impact <b>High</b></span>
+          <span class="m">Wow <b class="wow">★★★★</b></span>
+          <span class="m">BM25 hit@1 is 0.340 on the pinned corpus and 0.040 on a real one</span>
+        </div>
+      </article>
       <!-- /ROADMAP:cards -->
 
     </div>
@@ -4347,12 +4390,56 @@ about 1.2 s per chunk. This card's static-embedding proposal claims ~29 ms/doc i
 that holds it is a difference of more than an order of magnitude on the doc side, which is exactly
 the cost that makes prebuilt shards mandatory today. Worth measuring the model2vec path directly
 before committing &mdash; but the gap it claims to close is now a measured number rather than an
-estimate.</p>
+estimate.
+
+<b>Spike done &mdash; the prerequisites this card set have all shipped, so the measurement it asked
+for was finally runnable.</b> It says &ldquo;do not ship before the eval and dedup items&rdquo;;
+dedup landed in <code>#370</code>, the index format in <code>#367</code>/<code>#371</code>, the
+published eval in <code>#373</code>. What follows is <code>potion-retrieval-32M</code> (MIT, 63,091
+&times; 512 F32 lookup table &mdash; confirmed a single tensor, no transformer) driven by a
+hand-written pure-stdlib loader: WordPiece &rarr; gather rows &rarr; mean-pool &rarr; L2, mmap'd,
+no numpy.
+
+<b>The two unverified performance claims were not just right, they were conservative.</b> Query
+embedding measured <b>0.16&nbsp;ms</b> against the card's <code>~1&nbsp;ms</code>. Document
+embedding measured <b>1.34&nbsp;ms</b> on a synthetic 105-token doc and <b>3.27&nbsp;ms</b> on 300
+real catalogue entries (median 61 tokens), against the card's <code>~29&nbsp;ms</code>.
+
+<b>That reverses one of this card's design arguments.</b> The doc-side cost was the reason prebuilt
+artifacts looked mandatory: &ldquo;29&nbsp;ms/doc in pure Python is ~24&nbsp;min for 50k
+single-core&rdquo;. At the measured 3.27&nbsp;ms it is <b>2.7&nbsp;min</b> &mdash; roughly the time
+a first <code>boost tap --defaults</code> already takes. Local embedding is therefore viable on the
+user's own machine, and shipped shards become a genuine optimisation rather than a requirement. (Not
+to be confused with the ONNX <code>bge-small</code> path measured at ~1.2&nbsp;s/chunk in
+<code>keyless-semantic-search-for-everyone</code>; that number stands, and the gap between them
+<em>is</em> the case for the static model.)
+
+<b>But reranking bought nothing at real scale, which is the result that matters.</b> Over the 50
+natural-language golden queries against a real <b>71,655-entry</b> catalogue, reranking BM25's
+top-200 by cosine scored <code>hit@1</code> <b>2/50</b> &mdash; identical to BM25's own
+<b>2/50</b>, a net change of <b>+0 queries</b> where this card's own statistics note says 6 net
+queries is the smallest win reaching p&lt;0.05. On two hand-checked pairs the ordering was right but
+the margin was thin (related 0.154 vs unrelated 0.097).
+
+<b>Stated limits, because this does not settle the question.</b> The document vector was built from
+<code>name + description</code> truncated to 1,500 characters, not the full body the real dense path
+indexes, so this measures a weaker representation than the one being proposed. No blend was tried
+&mdash; pure rerank, no <code>w_dense</code> &mdash; and this card explicitly asks for a held-out
+blend weight and McNemar. What it does establish is that the <em>cheap</em> version of the idea does
+not pay for itself, so the remaining work is representation and blending, not inference speed.
+
+<b>An unrelated finding fell out of it, and it is the more important one.</b> BM25 scored
+<code>hit@1</code> <b>0.040</b> here against the <b>0.340</b> published in <code>#373</code>. Both
+are correct: the published figure is measured over the pinned 6-tap eval corpus of <b>743</b>
+entries, and this run used a real 77-tap install &mdash; <b>96&times; larger</b>. Golden targets are
+all present and rank 7th, 8th, 38th, 163rd rather than 1st. The eval corpus is not a scale model of
+a real install, and the gate's floors describe a catalogue two orders of magnitude smaller than the
+one users have. Tracked separately in [[eval-corpus-is-96x-smaller-than-a-real-install]].</p>
         <div class="meta">
           <span class="m">Complexity <b>L</b></span>
           <span class="m hi">Impact <b>High</b></span>
           <span class="m">Wow <b class="wow">★★★★★</b></span>
-          <span class="m">semantic search with no key, no billing, no deps</span>
+          <span class="m">spike done: the perf claims were 6-22x conservative, but rerank added +0 at real scale</span>
         </div>
       </article>