boost-skill-cli 1.2.53__tar.gz → 1.2.54__tar.gz

{boost_skill_cli-1.2.53 → boost_skill_cli-1.2.54}/CLAUDE.md

@@ -493,6 +493,22 @@ each agent's skills dir, and updating the lock file.
   published. A matrix entry is a **space-separated list**, split in Python
   inside the job, never by shell word-splitting.
 
+- **A shard run reuses what it already published, and the manifest is how.**
+  Every weekly run used to embed the whole catalogue (~9 job-hours) on
+  ephemeral runners with no memory of last week, although registries move
+  slowly. `shards.unchanged(manifest, commits)` returns the rows whose pinned
+  commit is the tap's commit *now*; the build job removes those taps before
+  `reindex --dense`, and the publish job fetches last week's manifest from the
+  release and carries their rows forward verbatim (`publish_shards.py manifest
+  --carry-forward`). Carry forward, not import-and-re-export: the assets are
+  still on the release byte for byte (`gh release upload --clobber` replaces,
+  never deletes), so re-exporting would re-upload ~300 MB of identical vectors
+  a week. Three refusals keep it honest — an empty local commit is a failed
+  clone, not a match; a manifest in another embedding space reuses nothing;
+  and a row is carried only when the job's commit and the manifest's agree. A
+  registry that is neither fresh nor unchanged drops out of the manifest rather
+  than accumulating forever. Its asset stays; its row returns when it does.
+
 - **A published shard is only importable while three things match, and every
   mismatch is silent if waved through.** `core/shards.py` fetches
   `manifest.json` from a rolling `shards-latest` **prerelease** on this repo —

{boost_skill_cli-1.2.53/boost_skill_cli.egg-info → boost_skill_cli-1.2.54}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.2.53
+Version: 1.2.54
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License-Expression: Apache-2.0

{boost_skill_cli-1.2.53 → boost_skill_cli-1.2.54}/boost_cli/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '1.2.53'
-__version_tuple__ = version_tuple = (1, 2, 53)
+__version__ = version = '1.2.54'
+__version_tuple__ = version_tuple = (1, 2, 54)
 
 __commit_id__ = commit_id = None

{boost_skill_cli-1.2.53 → boost_skill_cli-1.2.54}/boost_cli/core/shards.py

@@ -171,6 +171,33 @@ def rows(manifest: dict) -> dict[str, dict]:
     return out
 
 
+def unchanged(manifest: dict, commits: dict[str, str]) -> dict[str, dict]:
+    """Manifest rows that already describe the commit each tap is at now.
+
+    This is what lets a weekly shard run skip most of its own work. Every run
+    used to embed every registry from scratch — ~9 job-hours for the catalogue
+    — on ephemeral runners with no memory of last week, and registries move
+    slowly, so most of that bought the same vectors again. The manifest pins
+    the commit each shard was built from, so "has this one moved?" is one
+    comparison, and a registry that has not keeps last week's row.
+
+    Same rule that makes a shard importable at all: reuse only for the EXACT
+    commit the row describes. An empty local commit is a tap whose clone
+    failed, not a match — that registry is embedded, never skipped. The
+    embedding-space check is the caller's (:func:`incompatible`), because a
+    manifest from another space reuses nothing however fresh its commits.
+    """
+    index = rows(manifest)
+    out: dict[str, dict] = {}
+    for tap, commit in commits.items():
+        row = index.get(tap)
+        if row is None or not commit:
+            continue
+        if str(row.get("commit")) == commit:
+            out[tap] = row
+    return out
+
+
 def _same_origin(a: str, b: str) -> bool:
     """True when two URLs share scheme+host+port.
 

{boost_skill_cli-1.2.53 → boost_skill_cli-1.2.54/boost_skill_cli.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.2.53
+Version: 1.2.54
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License-Expression: Apache-2.0

{boost_skill_cli-1.2.53 → boost_skill_cli-1.2.54}/boost_skill_cli.egg-info/SOURCES.txt

@@ -439,6 +439,7 @@ docs/roadmap/items/self-update-claims-up-to-date-without-asking.md
 docs/roadmap/items/serve-404-reflected-the-request.md
 docs/roadmap/items/serve-is-a-searchable-catalogue.md
 docs/roadmap/items/serve-py-traversal-guards-untested.md
+docs/roadmap/items/shard-runs-reembed-everything-weekly.md
 docs/roadmap/items/shards-pipeline-never-produced-a-shard.md
 docs/roadmap/items/shareable-catalogue-bundle.md
 docs/roadmap/items/shift-left-gate-pre-commit-pre-commit-ci.md
@@ -783,6 +784,7 @@ tests/unit/test_serve.py
 tests/unit/test_serve_docs.py
 tests/unit/test_shard_plan.py
 tests/unit/test_shards.py
+tests/unit/test_shards_incremental.py
 tests/unit/test_shards_matrix.py
 tests/unit/test_smoke_covers_every_command.py
 tests/unit/test_spdx_headers.py

boost_skill_cli-1.2.54/docs/roadmap/items/shard-runs-reembed-everything-weekly.md

@@ -0,0 +1,48 @@
+---
+id: shard-runs-reembed-everything-weekly
+board: code
+section: pipeline
+status: shipped
+category: Retrieval · CI
+complexity: M
+impact: High
+wow: 4
+note: the manifest already knew which registries had not moved
+order: 2
+owner: loop/shards-incremental
+pr:
+title: Every weekly shard run re-embedded the whole catalogue
+---
+The catalogue shard run is <b>60 packed jobs over 463 registries, ~9 job-hours a week</b> at the
+measured 0.22&nbsp;s/chunk &mdash; and every one of those hours was spent on ephemeral runners with
+no memory of the week before. Registries move slowly. Most weeks most of the catalogue is at the
+same commit it was published at, and the run bought the same vectors again, byte for byte.
+
+<b>The manifest already held the answer.</b> Each row pins the registry commit its shard was
+built from, because <code>dense.import_shard</code> refuses a shard for any other commit. That
+same pin answers "has this registry moved since we published?" in one comparison, and
+<code>shards.unchanged(manifest, commits)</code> makes it: a tap is unchanged only for the
+<em>exact</em> commit its row describes, an empty local commit (a clone that failed) never
+counts, and a manifest in another embedding space reuses nothing however fresh its commits
+&mdash; none of its rows would be importable by the consumer this run publishes for.
+
+<b>Carry forward, not re-export.</b> The build job taps its chunk, asks
+<code>publish_shards.py unchanged</code>, untaps every registry it lists, and embeds only what
+is left &mdash; a chunk with nothing left skips the pass rather than failing on "no taps
+configured". The publish job fetches last week's manifest from the release and carries those
+rows forward verbatim: the assets are still there, since <code>gh release upload --clobber</code>
+replaces but never deletes. The cheaper-looking alternative, import last week's shard and
+re-export it, would re-upload ~300&nbsp;MB of identical vectors a week for nothing.
+
+<b>What the row set means now.</b> Fresh beats carried for the same tap. A registry that is
+neither fresh nor unchanged &mdash; removed from the catalogue, or failed to tap this week
+&mdash; drops out of the manifest rather than accumulating forever; its asset stays on the
+release, harmless, and its row returns the week the registry does. A row is carried only when
+the job's commit and the manifest's agree: two sources disagreeing means someone is wrong, and
+the honest outcome is a re-embed next run, not a row that may describe a tree the registry has
+left.
+
+<b>Cost after.</b> The first run is unchanged. Every run after it costs the registries that
+moved plus one comparison each for the rest, so a quiet week is a manifest upload and little
+else, and wall clock is bounded by the largest registry that actually changed rather than by
+the largest registry.

{boost_skill_cli-1.2.53 → boost_skill_cli-1.2.54}/docs/roadmap.html

@@ -133,7 +133,7 @@
       <div class="stat"><b>20</b><span>Shipped</span></div>
       <div class="stat"><b>1</b><span>Next up</span></div>
       <div class="stat"><b>7</b><span>Planned</span></div>
-      <div class="stat"><b>288</b><span>Loop finds</span></div>
+      <div class="stat"><b>289</b><span>Loop finds</span></div>
       <!-- /ROADMAP:stats -->
     </div>
   </section>
@@ -5704,6 +5704,51 @@ header claims rows only a Voyage key can read.</p></details>
         </div>
       </article>
 
+      <article class="cap rcard" id="shard-runs-reembed-everything-weekly">
+        <div class="head"><span class="pill shipped">Shipped</span><span class="cat">Retrieval · CI</span></div>
+        <h3>Every weekly shard run re-embedded the whole catalogue</h3>
+        <details class="cardbody"><summary>Write-up</summary>
+        <p>The catalogue shard run is <b>60 packed jobs over 463 registries, ~9 job-hours a week</b> at the
+measured 0.22&nbsp;s/chunk &mdash; and every one of those hours was spent on ephemeral runners with
+no memory of the week before. Registries move slowly. Most weeks most of the catalogue is at the
+same commit it was published at, and the run bought the same vectors again, byte for byte.
+
+<b>The manifest already held the answer.</b> Each row pins the registry commit its shard was
+built from, because <code>dense.import_shard</code> refuses a shard for any other commit. That
+same pin answers "has this registry moved since we published?" in one comparison, and
+<code>shards.unchanged(manifest, commits)</code> makes it: a tap is unchanged only for the
+<em>exact</em> commit its row describes, an empty local commit (a clone that failed) never
+counts, and a manifest in another embedding space reuses nothing however fresh its commits
+&mdash; none of its rows would be importable by the consumer this run publishes for.
+
+<b>Carry forward, not re-export.</b> The build job taps its chunk, asks
+<code>publish_shards.py unchanged</code>, untaps every registry it lists, and embeds only what
+is left &mdash; a chunk with nothing left skips the pass rather than failing on "no taps
+configured". The publish job fetches last week's manifest from the release and carries those
+rows forward verbatim: the assets are still there, since <code>gh release upload --clobber</code>
+replaces but never deletes. The cheaper-looking alternative, import last week's shard and
+re-export it, would re-upload ~300&nbsp;MB of identical vectors a week for nothing.
+
+<b>What the row set means now.</b> Fresh beats carried for the same tap. A registry that is
+neither fresh nor unchanged &mdash; removed from the catalogue, or failed to tap this week
+&mdash; drops out of the manifest rather than accumulating forever; its asset stays on the
+release, harmless, and its row returns the week the registry does. A row is carried only when
+the job's commit and the manifest's agree: two sources disagreeing means someone is wrong, and
+the honest outcome is a re-embed next run, not a row that may describe a tree the registry has
+left.
+
+<b>Cost after.</b> The first run is unchanged. Every run after it costs the registries that
+moved plus one comparison each for the rest, so a quiet week is a manifest upload and little
+else, and wall clock is bounded by the largest registry that actually changed rather than by
+the largest registry.</p></details>
+        <div class="meta">
+          <span class="m">Complexity <b>M</b></span>
+          <span class="m hi">Impact <b>High</b></span>
+          <span class="m">Wow <b class="wow">★★★★</b></span>
+          <span class="m">the manifest already knew which registries had not moved</span>
+        </div>
+      </article>
+
       <article class="cap rcard" id="workflow-linting-actionlint">
         <div class="head"><span class="pill shipped">Shipped</span><span class="cat">Quality · CI/CD</span></div>
         <h3>Workflow linting — <code>actionlint</code></h3>

boost_skill_cli-1.2.54/scripts/publish_shards.py

@@ -0,0 +1,303 @@
+#!/usr/bin/env python3
+# Copyright the boost contributors.
+# SPDX-License-Identifier: Apache-2.0
+"""Turn exported vector shards into something a stranger can download.
+
+Three subcommands, matching the three halves of publishing:
+
+    publish_shards.py export --out DIR [--tap owner/repo ...]
+        Export every tapped registry's vectors from THIS machine's store.
+        Run it where the embeddings already exist — that is the whole point of
+        a shard, and re-embedding to publish would defeat it.
+
+    publish_shards.py unchanged --manifest-url URL --out FILE
+        Which tapped registries are already published at the commit they are
+        at now. Run it after tapping and BEFORE embedding: every registry it
+        lists can be untapped, and the embed pass costs only what moved.
+
+    publish_shards.py manifest --shard-dir DIR --repo owner/repo [--tag TAG]
+                              [--carry-forward PREV.json --unchanged FILE...]
+        Digest the shards in DIR and write the manifest that `core.shards`
+        reads: schema version, the embedding space they all share, and one row
+        per shard with its registry commit, size, sha256 and download URL. With
+        `--carry-forward`, rows from the previous manifest survive for the
+        registries the build jobs reported unchanged — their assets are still
+        on the release, byte for byte, so the old row is the right row.
+
+WHY THE MANIFEST IS GENERATED AND NOT WRITTEN BY HAND. Three of its fields are
+load-bearing at import time and unguessable: `sha256` is what makes a download
+verifiable, `commit` is what stops a stale shard being merged, and
+`provider`/`model`/`dim` are what stop vectors from two different embedding
+spaces being mixed into one store — a failure that does not raise, it just
+returns nonsense rankings.
+
+WHY CARRY FORWARD RATHER THAN RE-EXPORT. Every weekly run used to embed every
+registry from scratch (~9 job-hours for the catalogue) although most registries
+had not moved. The cheap alternative — import last week's shard and re-export
+it — re-uploads ~300 MB of identical vectors a week. Carrying the row forward
+uploads nothing: the asset is already on the release, and `--clobber` never
+deletes what it does not replace.
+
+WHICH EMBEDDING SPACE TO PUBLISH. The keyless one. `embed` resolves Voyage, then
+OpenAI, then the local ONNX `BAAI/bge-small-en-v1.5` that ships with the `rag`
+extra, so a shard built on a machine holding `VOYAGE_API_KEY` is 1024-d
+`voyage-4` and can only be imported — and queried — by someone else holding that
+key. Export refuses to mix spaces for that reason, and says which one it found.
+"""
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import sys
+from datetime import UTC, datetime
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+
+from boost_cli.core import dense, gitutil, registry, shards
+from boost_cli.errors import BoostError
+
+DEFAULT_TAG = "shards-latest"
+
+
+def _safe(tap: str) -> str:
+    return tap.replace("/", "__")
+
+
+def _space(d: dict) -> dict:
+    """The (provider, model, dim) triple, normalised so JSON and shard agree."""
+    return {"provider": str(d.get("provider") or ""),
+            "model": str(d.get("model") or ""),
+            "dim": int(d.get("dim") or 0)}
+
+
+def _fmt_space(space: dict) -> str:
+    return "%s/%s/%sd" % (space["provider"], space["model"], space["dim"])
+
+
+def cmd_export(args: argparse.Namespace) -> int:
+    """Write one `<tap>.shard.json` per tap into `--out`."""
+    taps = args.tap or [t.name for t in registry.list_taps()]
+    if not taps:
+        print("no taps configured on this machine", file=sys.stderr)
+        return 1
+    out_dir = Path(args.out)
+    out_dir.mkdir(parents=True, exist_ok=True)
+    written = 0
+    for tap in taps:
+        try:
+            shard = dense.export_shard(tap)
+        except BoostError as exc:
+            print("skip %s: %s" % (tap, exc.message), file=sys.stderr)
+            continue
+        if not shard.get("chunks"):
+            print("skip %s: no vectors" % tap, file=sys.stderr)
+            continue
+        dest = out_dir / (_safe(tap) + ".shard.json")
+        dest.write_text(json.dumps(shard), encoding="utf-8")
+        written += 1
+        print("%s: %d chunks, %s @ %s"
+              % (tap, len(shard["chunks"]), shard.get("model"),
+                 str(shard.get("commit"))[:8]))
+    print("wrote %d shard(s) to %s" % (written, out_dir))
+    return 0 if written else 1
+
+
+def cmd_unchanged(args: argparse.Namespace) -> int:
+    """Write `tap commit` for every tap whose published shard is current.
+
+    Exit 0 whatever happens: an empty list means "embed everything", which is
+    the right outcome for a first run, an unreachable release, or a manifest in
+    another embedding space — and a workflow that aborted on any of those would
+    never publish the first shard.
+    """
+    lines: list[str] = []
+    try:
+        manifest = shards.fetch_manifest(args.manifest_url)
+    except BoostError as exc:
+        print("no usable manifest (%s) — embedding everything" % exc.message,
+              file=sys.stderr)
+        manifest = None
+    if manifest is not None:
+        why = shards.incompatible(manifest)
+        if why:
+            print("published shards are not in this run's space (%s) — "
+                  "embedding everything" % why, file=sys.stderr)
+        else:
+            commits = {t.name: (gitutil.head_commit(t.path) if t.is_cloned
+                                else "")
+                       for t in registry.list_taps()}
+            for tap, row in shards.unchanged(manifest, commits).items():
+                lines.append("%s %s" % (tap, row["commit"]))
+    text = "".join(line + "\n" for line in lines)
+    if args.out:
+        Path(args.out).write_text(text, encoding="utf-8")
+    else:
+        sys.stdout.write(text)
+    print("%d registr%s unchanged since the last publish"
+          % (len(lines), "y" if len(lines) == 1 else "ies"), file=sys.stderr)
+    return 0
+
+
+def _fresh_row(path: Path, repo: str, tag: str) -> tuple[dict, dict]:
+    """One manifest row plus the embedding space its shard was built in."""
+    raw = path.read_bytes()
+    shard = json.loads(raw.decode("utf-8"))
+    missing = [k for k in ("tap", "commit", "provider", "model", "dim")
+               if not shard.get(k)]
+    if missing:
+        raise SystemExit("%s is missing %s" % (path.name, ", ".join(missing)))
+    row = {
+        "tap": shard["tap"],
+        "commit": shard["commit"],
+        "chunks": len(shard.get("chunks") or []),
+        "bytes": len(raw),
+        "sha256": hashlib.sha256(raw).hexdigest(),
+        "url": "https://github.com/%s/releases/download/%s/%s"
+               % (repo, tag, path.name),
+    }
+    return row, _space(shard)
+
+
+def _carried_rows(prev_path: str, unchanged_files: list[str],
+                  fresh_taps: set[str], space: dict | None
+                  ) -> tuple[list[dict], dict | None]:
+    """Rows from the previous manifest for registries reported unchanged.
+
+    Returns ``(rows, previous_space)``. Every guard here refuses rather than
+    degrades: a row is carried only for the exact commit the job saw, only
+    when the previous manifest is in this run's embedding space, and never
+    for a registry a fresh shard already describes.
+    """
+    try:
+        prev = json.loads(Path(prev_path).read_text(encoding="utf-8"))
+    except (OSError, json.JSONDecodeError) as exc:
+        print("previous manifest unreadable (%s) — carrying nothing forward"
+              % exc, file=sys.stderr)
+        return [], None
+    prev_space = _space(prev)
+    if space is not None and prev_space != space:
+        print("previous manifest is %s, this run is %s — carrying nothing "
+              "forward" % (_fmt_space(prev_space), _fmt_space(space)),
+              file=sys.stderr)
+        return [], prev_space
+    wanted: dict[str, str] = {}
+    for name in unchanged_files:
+        try:
+            text = Path(name).read_text(encoding="utf-8")
+        except OSError as exc:
+            print("cannot read %s (%s) — its registries will be re-embedded "
+                  "next run" % (name, exc), file=sys.stderr)
+            continue
+        for line in text.splitlines():
+            parts = line.split()
+            if len(parts) == 2:
+                wanted[parts[0]] = parts[1]
+    index = shards.rows(prev)
+    out: list[dict] = []
+    for tap, commit in sorted(wanted.items()):
+        if tap in fresh_taps:
+            continue
+        row = index.get(tap)
+        if row is None or str(row.get("commit")) != commit:
+            # The job says one commit, the manifest another: someone is wrong,
+            # and a row carried under those conditions could describe a tree
+            # the registry is no longer at. Re-embedded next run.
+            continue
+        out.append(dict(row))
+    return out, prev_space
+
+
+def cmd_manifest(args: argparse.Namespace) -> int:
+    """Digest a directory of shards (plus carried rows) into one manifest."""
+    shard_dir = Path(args.shard_dir)
+    files = sorted(shard_dir.glob("*.shard.json"))
+    rows: list[dict] = []
+    spaces: list[dict] = []
+    for path in files:
+        row, space = _fresh_row(path, args.repo, args.tag)
+        rows.append(row)
+        spaces.append(space)
+    first = spaces[0] if spaces else None
+    for path, space in zip(files, spaces, strict=True):
+        if space != first:
+            # Refuse rather than publish a manifest whose top-level space is a
+            # lie for some of its rows: `core.shards.incompatible` reads that
+            # one header to decide whether to download anything at all.
+            raise SystemExit(
+                "%s is %s but the others are %s — publish one embedding space "
+                "per manifest" % (path.name, _fmt_space(space),
+                                  _fmt_space(first or {})))
+    carried = 0
+    if args.carry_forward:
+        old, prev_space = _carried_rows(args.carry_forward, args.unchanged,
+                                        {r["tap"] for r in rows}, first)
+        rows.extend(old)
+        carried = len(old)
+        if first is None and old:
+            # A quiet week: nothing moved, nothing fresh, and the manifest is
+            # still due. Its space is the previous manifest's.
+            first = prev_space
+    if not rows or first is None:
+        print("nothing to publish: no *.shard.json under %s and nothing "
+              "carried forward" % shard_dir, file=sys.stderr)
+        return 1
+    manifest = {
+        "version": 1,
+        "generated": datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%SZ"),
+        "provider": first["provider"],
+        "model": first["model"],
+        "dim": first["dim"],
+        "shards": sorted(rows, key=lambda r: r["tap"]),
+    }
+    dest = Path(args.out)
+    dest.write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8")
+    total = sum(int(r.get("chunks") or 0) for r in rows)
+    print("manifest: %d shard(s) (%d fresh, %d carried forward), %s chunks, "
+          "%s -> %s" % (len(rows), len(rows) - carried, carried,
+                        format(total, ","), _fmt_space(first), dest))
+    return 0
+
+
+def main(argv: list[str] | None = None) -> int:
+    p = argparse.ArgumentParser(description=__doc__,
+                                formatter_class=argparse.RawDescriptionHelpFormatter)
+    sub = p.add_subparsers(dest="cmd", required=True)
+
+    ex = sub.add_parser("export", help="export this machine's vectors")
+    ex.add_argument("--out", required=True, metavar="DIR")
+    ex.add_argument("--tap", action="append", metavar="OWNER/REPO",
+                    help="only this tap (repeatable); default is every tap")
+    ex.set_defaults(func=cmd_export)
+
+    un = sub.add_parser("unchanged",
+                        help="list tapped registries already published at "
+                             "their current commit")
+    un.add_argument("--manifest-url", metavar="URL",
+                    help="the published manifest (default: boost's own)")
+    un.add_argument("--out", metavar="FILE",
+                    help="write `tap commit` lines here instead of stdout")
+    un.set_defaults(func=cmd_unchanged)
+
+    mf = sub.add_parser("manifest", help="write manifest.json for a shard dir")
+    mf.add_argument("--shard-dir", required=True, metavar="DIR")
+    mf.add_argument("--repo", required=True, metavar="OWNER/REPO",
+                    help="repo whose release hosts the assets")
+    mf.add_argument("--tag", default=DEFAULT_TAG,
+                    help="release tag hosting the assets (default: %(default)s)")
+    mf.add_argument("--out", default="manifest.json", metavar="FILE")
+    mf.add_argument("--carry-forward", metavar="PREV.json",
+                    help="the previous manifest; its rows survive for "
+                         "registries listed in --unchanged files")
+    mf.add_argument("--unchanged", nargs="*", default=[], metavar="FILE",
+                    help="`tap commit` files written by "
+                         "`publish_shards.py unchanged` in the build jobs")
+    mf.set_defaults(func=cmd_manifest)
+
+    args = p.parse_args(argv)
+    return int(args.func(args))
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())