boost-skill-cli 1.2.63__tar.gz → 1.2.65__tar.gz

{boost_skill_cli-1.2.63 → boost_skill_cli-1.2.65}/.gitignore

@@ -281,3 +281,10 @@ node_modules/
 # config). Written into the repo root by the language-server tooling, not by
 # boost, and specific to one developer's machine.
 .serena/
+
+# A SQLite artifact from a tool that passed ":memory:" as a path rather than as
+# the in-memory sentinel. Junk either way, and on Windows it is worse than junk:
+# a colon cannot appear in a filename there, so committing it makes `git
+# checkout` fail with exit 128 and every Windows CI job dies before it runs a
+# single test.
+:memory:*

{boost_skill_cli-1.2.63 → boost_skill_cli-1.2.65}/CLAUDE.md

@@ -653,6 +653,55 @@ Consequences for code you write here:
   the whole table instead measured 3.75 s cold on a 657,587-chunk store, and
   ran even on the build where nothing had changed.
 
+- **The store holds one vector per distinct embedding, and `chunks.vid` is the
+  indirection.** Every chunk row used to own its own `vec_raw` blob and its own
+  `vec_chunks_bin` row, so a paragraph vendored into 1,464 skills was stored
+  1,464 times. Measured on a real 657,587-chunk / 384-d store: the vector rows
+  collapse to **396,638 distinct** (39.7% repeats), vectors are 81.6% of the
+  1,634 MB file, and sharing them is **1,634 → ~1,140 MB, 1.43x whole-store**.
+  `chunks` still gets one row per copy — tap deletion is scoped through
+  `chunks.tap`, so collapsing rows would strand a tap's vectors — and names its
+  vector through `vid`; `vectors(vid, hash)` is the relation, and `vec_raw` /
+  `vec_chunks_bin` / `vec_chunks` are keyed by `vid`. Four rules:
+  - **The key is `sha256(embedding)`, never the text.** `export_shard` ships
+    `snip` (`text[:200]`), so the full chunk text is in neither the store nor
+    the shard and `import_shard` has nothing to hash — a `text_hash` column
+    could never be filled in on the import path. The blob is equivalent by
+    construction (a provider is deterministic) and answers the same question
+    for a local build, an imported shard *and* the migration.
+  - **Deletion collects the freed vectors after the row delete, never
+    before.** `_delete_matching` used to drop vectors then rows, which was
+    right while they were one-to-one. Asking first counts references that the
+    very next statement removes, so every shared vector reads as live and
+    survives for good — nothing later revisits it, because every sweep here is
+    scoped through `chunks`. The answer is derived (`_gc_vectors`, one indexed
+    probe per candidate through `chunks_vid`) rather than stored in a counter,
+    so it cannot drift out of step with the rows it describes.
+  - **Migration is in place, not an `INDEX_VERSION` bump**, following
+    `quantize()`'s precedent — v3's bump was meant to be the last re-embed and
+    reclaiming disk is not a reason to break that. `_adopt_vectors` runs inside
+    `_ensure_schema` and is *structural only*: `vid = id` is the identity those
+    numbers already had, so it reads no blobs. `deduplicate()` is the expensive
+    half — `boost reindex --dense` runs it **after** `quantize()`, because
+    quantize moves vectors one `rowid` at a time and refuses a store whose
+    vector rows no longer match its chunk rows. The survivor of a cluster is
+    its lowest id, so nothing is rewritten: the copies are deleted and
+    `chunks.vid` redirected.
+  - **A NULL `hash` is never a match.** Adopted rows carry no hash because
+    nothing hashed them, and `vectors.hash` is UNIQUE *and* accepts NULL, so
+    SQLite holds those unknowns apart. A build stores its own copy rather than
+    claiming a vector it never compared — a missed saving `deduplicate()`
+    reclaims, against a wrong answer nothing would notice. That UNIQUE index is
+    also what makes "already done" cheap: two hashed rows provably cannot hold
+    the same bytes, so one `hash IS NULL` probe decides whether the pass has
+    anything to do.
+  Reuse is now cross-build and cross-tap: `_embed_and_store`'s `seen` dict only
+  ever collapsed copies inside one run, so a newly tapped mirror registry cost
+  a full set of vectors for content already on disk. The invariant to assert is
+  **not** a `chunks.id` ↔ `vec_raw.id` bijection (sharing breaks one direction
+  by design) but the pair `tests/unit/test_dense_vector_dedupe.py` pins: every
+  `chunks.vid` resolves, and every vector has at least one referent.
+
 - **Nothing on the search path may count `chunks`.** `SELECT COUNT(*)` scans the
   `chunks_tap` covering index — 8,419 pages / 34.5 MB, measured 1.94 s — and
   both `ready()` and `status()` did it on every search, the latter only to word

{boost_skill_cli-1.2.63/boost_skill_cli.egg-info → boost_skill_cli-1.2.65}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.2.63
+Version: 1.2.65
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License-Expression: Apache-2.0

{boost_skill_cli-1.2.63 → boost_skill_cli-1.2.65}/boost_cli/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '1.2.63'
-__version_tuple__ = version_tuple = (1, 2, 63)
+__version__ = version = '1.2.65'
+__version_tuple__ = version_tuple = (1, 2, 65)
 
 __commit_id__ = commit_id = None

{boost_skill_cli-1.2.63 → boost_skill_cli-1.2.65}/boost_cli/commands/discovery.py

@@ -493,6 +493,11 @@ def cmd_reindex(argv):
                 out.ok("quantized %d existing vectors — dense search no longer "
                        "scans the whole store on every query"
                        % dense_stats["quantized"])
+            if dense_stats.get("deduplicated"):
+                out.ok("reclaimed %d duplicate vector%s — registries paste the "
+                       "same text, and the store now holds one copy of each"
+                       % (dense_stats["deduplicated"],
+                          "" if dense_stats["deduplicated"] == 1 else "s"))
     journal.log("reindex", "%d passages" % stats["docs"],
                 entries=stats["entries"])
     return 0
@@ -548,9 +553,18 @@ def _reindex_dense(force, spinner=None):
     # only what changed. Doing it after would quantize, then write float32 rows
     # into a store that no longer has a float32 table.
     migrated = dense.quantize()
+    # Then collapse the byte-identical copies, in that order and not the other:
+    # `quantize` moves vectors out of `vec_chunks` one rowid at a time and
+    # would refuse a store whose vector rows no longer match its chunk rows,
+    # which is exactly what deduplicating first produces. Both are offline and
+    # free — they re-encode and re-key what is already on disk — so a user gets
+    # the disk back without having to know either word.
+    collapsed = dense.deduplicate()
     stats = dense.build(force=force, on_progress=_embed_progress(spinner))
     if migrated and isinstance(stats, dict):
         stats = stats | {"quantized": migrated["chunks"]}
+    if collapsed and isinstance(stats, dict) and collapsed["freed"]:
+        stats = stats | {"deduplicated": collapsed["freed"]}
     return stats