ossllms 0.1.0__tar.gz

ossllms-0.1.0/.gitignore

@@ -0,0 +1,49 @@
+# Secrets & keys — NEVER commit signing keys
+*.key
+*.pem
+*.sec
+*.minisign
+minisign.key
+*.private
+secrets/
+.env
+.env.*
+
+# Model data / large artifacts (these live in the swarm, not git)
+data/
+cache/
+e2e/artifacts/
+*.safetensors
+*.gguf
+*.bin
+*.pt
+*.ckpt
+*.torrent
+*.sqlite
+*.sqlite-*
+
+# Curated package data is intentionally tiny and must ship in the wheel.
+!sdk/ossllms/data/
+!sdk/ossllms/data/manifest.schema.json
+!sdk/ossllms/data/default-catalog/
+!sdk/ossllms/data/default-catalog/**
+!sdk/ossllms/data/trusted-keys/
+!sdk/ossllms/data/trusted-keys/**
+
+# Build / deps
+node_modules/
+dist/
+build/
+target/
+__pycache__/
+*.pyc
+.venv/
+venv/
+*.egg-info/
+
+# OS / editor
+.DS_Store
+Thumbs.db
+.idea/
+.vscode/
+*.swp

ossllms-0.1.0/PKG-INFO

@@ -0,0 +1,120 @@
+Metadata-Version: 2.4
+Name: ossllms
+Version: 0.1.0
+Summary: Pull open-weight AI models from the ossllms torrent network. A drop-in for huggingface_hub.
+Project-URL: Homepage, https://ossllms.com
+Project-URL: Source, https://github.com/gittb/ossllms
+Author: ossllms contributors
+License-Expression: Apache-2.0
+Keywords: ai,bittorrent,huggingface,llm,p2p,preservation
+Requires-Python: >=3.10
+Requires-Dist: cryptography>=42
+Provides-Extra: dev
+Requires-Dist: jsonschema>=4; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Provides-Extra: live-e2e
+Requires-Dist: huggingface-hub>=0.23; extra == 'live-e2e'
+Provides-Extra: schema
+Requires-Dist: jsonschema>=4; extra == 'schema'
+Provides-Extra: torrent
+Requires-Dist: libtorrent>=2.0; extra == 'torrent'
+Description-Content-Type: text/markdown
+
+# ossllms (Python SDK + CLI)
+
+Pull open-weight AI models from the [ossllms](https://ossllms.com) preservation
+network. A drop-in for `huggingface_hub` that resolves from the torrent layer,
+verifies integrity + publisher signature, and lands files in the HF-compatible
+cache. Full design: [`../docs/SDK.md`](../docs/SDK.md) and
+[`../docs/INTEROP-HF.md`](../docs/INTEROP-HF.md).
+
+## Install
+
+```bash
+pip install 'ossllms[torrent]' # V0 path: HTTP web-seed + libtorrent swarm/seed
+```
+
+## CLI (simplest UI)
+
+```bash
+ossllms search minilm                 # find a useful default-catalog model
+ossllms pull hf/sentence-transformers/all-MiniLM-L6-v2@0.0.1+1110a243fdf4
+ossllms pull hf/sentence-transformers/all-MiniLM-L6-v2@0.0.1+1110a243fdf4 --seed
+ossllms contribute                    # start/attach to the managed seed worker
+ossllms pull sha256:<64-hex>          # resolve by catalog hash index
+ossllms publish ./model               # TTY: prompt/infer, sign, optionally seed
+ossllms publish ./model --seed --yes-public # automation: publish and seed local bytes
+ossllms contribute --dry-run          # preview seed/publish candidates and caps
+ossllms contribute --publish          # TTY: choose publish/seed rows, consent, then seed
+ossllms contribute --publish --yes-public # automation: publish selected cache candidates, then seed
+ossllms contribute                    # attach to/start managed seed handoff worker
+ossllms contribute --status           # show worker metrics
+ossllms contribute --stop             # stop worker
+ossllms ls                            # what's cached
+ossllms verify hf/sentence-transformers/all-MiniLM-L6-v2@0.0.1+1110a243fdf4
+```
+
+If no catalog is configured, the packaged signed default catalog is used. Set
+`OSSLLMS_CATALOG=https://catalog.ossllms.com` or pass `--catalog ...` for a
+custom directory or http(s) catalog.
+
+## Python
+
+```python
+from ossllms import snapshot_download, hf_hub_download, pull
+
+path = snapshot_download("hf/sentence-transformers/all-MiniLM-L6-v2@0.0.1+1110a243fdf4")
+cfg  = hf_hub_download(
+    "hf/sentence-transformers/all-MiniLM-L6-v2",
+    "config.json",
+    revision="0.0.1+1110a243fdf4",
+)
+
+res = pull("hf/sentence-transformers/all-MiniLM-L6-v2@0.0.1+1110a243fdf4")
+print(res.verdict.label, res.verdict.signer)             # "Verified" / signer
+
+import ossllms.compat   # opt-in: route huggingface_hub downloads through ossllms
+```
+
+## What's built (v0.1)
+
+- `pull`: resolve signed manifest → download via **HTTP web seeds** (works today;
+  `file://` supported) → verify **every file's SHA-256**, **v2 root**, and
+  **minisign/Ed25519** signature → place in HF cache. Integrity always enforced;
+  origin shown as Verified/Unverified (`--require-signature` to enforce).
+  `pull --seed` starts the managed contribution worker for the pulled snapshot
+  when the catalog has matching torrent metadata.
+- Store-qualified refs, `sha256:` refs, and catalog-paired `magnet:` refs.
+- `publish`: build and sign a public redistribution bundle from a local model
+  directory; optionally update a static V0 catalog/hash index with
+  `--catalog-dir`, and start the managed seed worker directly from the local
+  directory with `--seed`.
+- `contribute --dry-run`: scans local HF cache roots, matches catalog hashes,
+  shows seed/publish candidates, public-publish warnings, and upload caps.
+- `contribute`: attaches to an active managed worker or starts one for complete
+  in-network seed matches, persisting upload caps, zero-download seed metrics, and
+  a worker plan. In a terminal it renders the scan plan, lets the user select
+  publish/seed rows, shows caps, and requires public redistribution consent
+  before publish writes. When `ossllms[torrent]` is installed and
+  `release.torrent` metadata is available beside `release.json`, the worker
+  starts libtorrent seed mode from a hardlink-only view of the HF cache.
+- Worker-state, HF metadata, provenance, and seed-mode handoff primitives.
+- `huggingface_hub`-compatible `snapshot_download` / `hf_hub_download` + `compat` shim.
+- Manifest schema validation, trust store (pinned keys), selective `--include`.
+- libtorrent swarm engine: download scaffold plus live contribute seed adapter
+  behind the optional torrent extra. Local two-peer E2E gates prove fixture,
+  direct-publish, and contribute-publish no-web-seed swarms over libtorrent.
+- Bundled default MiniLM metadata includes `release.torrent`, so the post-pull
+  default contribution path can start live seed mode when `ossllms[torrent]` is
+  installed.
+
+## Develop / test
+
+```bash
+python -m venv .venv
+.venv/bin/pip install -e '.[dev]'
+.venv/bin/pytest          # or: PYTHONPATH=. .venv/bin/pytest tests
+```
+
+`Verified` = verified **origin + integrity** (+ a `matches HF` badge). It does
+**not** mean the weights are safe to run.

ossllms-0.1.0/README.md

@@ -0,0 +1,98 @@
+# ossllms (Python SDK + CLI)
+
+Pull open-weight AI models from the [ossllms](https://ossllms.com) preservation
+network. A drop-in for `huggingface_hub` that resolves from the torrent layer,
+verifies integrity + publisher signature, and lands files in the HF-compatible
+cache. Full design: [`../docs/SDK.md`](../docs/SDK.md) and
+[`../docs/INTEROP-HF.md`](../docs/INTEROP-HF.md).
+
+## Install
+
+```bash
+pip install 'ossllms[torrent]' # V0 path: HTTP web-seed + libtorrent swarm/seed
+```
+
+## CLI (simplest UI)
+
+```bash
+ossllms search minilm                 # find a useful default-catalog model
+ossllms pull hf/sentence-transformers/all-MiniLM-L6-v2@0.0.1+1110a243fdf4
+ossllms pull hf/sentence-transformers/all-MiniLM-L6-v2@0.0.1+1110a243fdf4 --seed
+ossllms contribute                    # start/attach to the managed seed worker
+ossllms pull sha256:<64-hex>          # resolve by catalog hash index
+ossllms publish ./model               # TTY: prompt/infer, sign, optionally seed
+ossllms publish ./model --seed --yes-public # automation: publish and seed local bytes
+ossllms contribute --dry-run          # preview seed/publish candidates and caps
+ossllms contribute --publish          # TTY: choose publish/seed rows, consent, then seed
+ossllms contribute --publish --yes-public # automation: publish selected cache candidates, then seed
+ossllms contribute                    # attach to/start managed seed handoff worker
+ossllms contribute --status           # show worker metrics
+ossllms contribute --stop             # stop worker
+ossllms ls                            # what's cached
+ossllms verify hf/sentence-transformers/all-MiniLM-L6-v2@0.0.1+1110a243fdf4
+```
+
+If no catalog is configured, the packaged signed default catalog is used. Set
+`OSSLLMS_CATALOG=https://catalog.ossllms.com` or pass `--catalog ...` for a
+custom directory or http(s) catalog.
+
+## Python
+
+```python
+from ossllms import snapshot_download, hf_hub_download, pull
+
+path = snapshot_download("hf/sentence-transformers/all-MiniLM-L6-v2@0.0.1+1110a243fdf4")
+cfg  = hf_hub_download(
+    "hf/sentence-transformers/all-MiniLM-L6-v2",
+    "config.json",
+    revision="0.0.1+1110a243fdf4",
+)
+
+res = pull("hf/sentence-transformers/all-MiniLM-L6-v2@0.0.1+1110a243fdf4")
+print(res.verdict.label, res.verdict.signer)             # "Verified" / signer
+
+import ossllms.compat   # opt-in: route huggingface_hub downloads through ossllms
+```
+
+## What's built (v0.1)
+
+- `pull`: resolve signed manifest → download via **HTTP web seeds** (works today;
+  `file://` supported) → verify **every file's SHA-256**, **v2 root**, and
+  **minisign/Ed25519** signature → place in HF cache. Integrity always enforced;
+  origin shown as Verified/Unverified (`--require-signature` to enforce).
+  `pull --seed` starts the managed contribution worker for the pulled snapshot
+  when the catalog has matching torrent metadata.
+- Store-qualified refs, `sha256:` refs, and catalog-paired `magnet:` refs.
+- `publish`: build and sign a public redistribution bundle from a local model
+  directory; optionally update a static V0 catalog/hash index with
+  `--catalog-dir`, and start the managed seed worker directly from the local
+  directory with `--seed`.
+- `contribute --dry-run`: scans local HF cache roots, matches catalog hashes,
+  shows seed/publish candidates, public-publish warnings, and upload caps.
+- `contribute`: attaches to an active managed worker or starts one for complete
+  in-network seed matches, persisting upload caps, zero-download seed metrics, and
+  a worker plan. In a terminal it renders the scan plan, lets the user select
+  publish/seed rows, shows caps, and requires public redistribution consent
+  before publish writes. When `ossllms[torrent]` is installed and
+  `release.torrent` metadata is available beside `release.json`, the worker
+  starts libtorrent seed mode from a hardlink-only view of the HF cache.
+- Worker-state, HF metadata, provenance, and seed-mode handoff primitives.
+- `huggingface_hub`-compatible `snapshot_download` / `hf_hub_download` + `compat` shim.
+- Manifest schema validation, trust store (pinned keys), selective `--include`.
+- libtorrent swarm engine: download scaffold plus live contribute seed adapter
+  behind the optional torrent extra. Local two-peer E2E gates prove fixture,
+  direct-publish, and contribute-publish no-web-seed swarms over libtorrent.
+- Bundled default MiniLM metadata includes `release.torrent`, so the post-pull
+  default contribution path can start live seed mode when `ossllms[torrent]` is
+  installed.
+
+## Develop / test
+
+```bash
+python -m venv .venv
+.venv/bin/pip install -e '.[dev]'
+.venv/bin/pytest          # or: PYTHONPATH=. .venv/bin/pytest tests
+```
+
+`Verified` = verified **origin + integrity** (+ a `matches HF` badge). It does
+**not** mean the weights are safe to run.

ossllms-0.1.0/ossllms/__init__.py

@@ -0,0 +1,33 @@
+"""ossllms — pull open-weight AI models from the ossllms preservation network.
+
+A drop-in for huggingface_hub that resolves from the torrent layer, verifies
+integrity + publisher signature, and lands files in the HF-compatible cache.
+
+    from ossllms import snapshot_download
+    path = snapshot_download("Qwen/Qwen2.5-7B")
+"""
+from __future__ import annotations
+
+from .api import (
+    IntegrityError,
+    PullResult,
+    UnverifiedOriginError,
+    hf_hub_download,
+    pull,
+    snapshot_download,
+)
+from .verify import TrustStore, Verdict
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "pull",
+    "snapshot_download",
+    "hf_hub_download",
+    "PullResult",
+    "Verdict",
+    "TrustStore",
+    "IntegrityError",
+    "UnverifiedOriginError",
+    "__version__",
+]

ossllms-0.1.0/ossllms/api.py

@@ -0,0 +1,194 @@
+"""High-level API. `pull()` is the core; `snapshot_download` / `hf_hub_download`
+mirror the huggingface_hub names for drop-in use.
+"""
+from __future__ import annotations
+
+import shutil
+import tempfile
+from dataclasses import dataclass
+from pathlib import Path, PurePosixPath
+from typing import Callable, List, Optional
+
+from . import config
+from .cache import place_into_cache, reuse_from_blobs
+from .denylist import load_denylist
+from .engine import EngineError, get_engine
+from .manifest import model_id, version
+from .resolve import resolve_manifest_with_source
+from .verify import TrustStore, Verdict, verify_manifest
+
+
+class IntegrityError(RuntimeError):
+    pass
+
+
+class UnverifiedOriginError(RuntimeError):
+    pass
+
+
+@dataclass
+class PullResult:
+    path: Path
+    repo_id: str
+    revision: str
+    manifest: dict
+    verdict: Verdict
+
+
+def pull(
+    ref: str,
+    *,
+    catalog: Optional[str] = None,
+    cache_dir=None,
+    revision: Optional[str] = None,
+    include: Optional[List[str]] = None,
+    engine: str = "auto",
+    engine_impl=None,
+    trust: Optional[TrustStore] = None,
+    denylist: Optional[str] = None,
+    require_signature: bool = False,
+    progress: Optional[Callable[[str], None]] = None,
+) -> PullResult:
+    """Resolve, download, verify, and cache a model version.
+
+    Integrity (per-file SHA-256) is ALWAYS enforced. Signature/origin failure
+    raises only when require_signature=True; otherwise it returns an Unverified
+    verdict (community upload) and proceeds, matching the trust UX.
+    """
+    catalog = catalog or config.default_catalog()
+    cache_dir = Path(cache_dir) if cache_dir else config.hf_hub_cache()
+    trust = trust if trust is not None else TrustStore.from_dir(config.trust_dir())
+
+    resolution = resolve_manifest_with_source(ref, catalog)
+    manifest = resolution.manifest
+    repo_id = model_id(manifest)
+    ver = version(manifest)
+    rev = revision or ver
+    denylist = denylist or config.default_denylist()
+    if denylist is not None:
+        load_denylist(denylist, trust).check_manifest(manifest)
+
+    with tempfile.TemporaryDirectory(prefix="ossllms-") as staged:
+        selected = include_to_paths(manifest, include)
+        for path in selected:
+            _safe_artifact_relpath(path)
+        sel_set = set(selected)
+        selected_artifacts = [a for a in manifest["artifacts"] if a["path"] in sel_set]
+
+        # Reuse byte-identical files already in the blob store; fetch only the rest.
+        reused = set(reuse_from_blobs(staged, cache_dir, repo_id, selected_artifacts))
+        missing = [p for p in selected if p not in reused]
+        local = set()
+        if resolution.local_source_dir is not None and missing:
+            local = set(
+                _copy_from_local_source(
+                    resolution.local_source_dir,
+                    staged,
+                    missing,
+                )
+            )
+            missing = [p for p in missing if p not in local]
+
+        eng = None
+        if missing:
+            eng = engine_impl if engine_impl is not None else get_engine(engine, manifest)
+        if progress:
+            engine_name = eng.name if eng is not None else "none"
+            progress(
+                f"engine: {engine_name}; reuse {len(reused)} file(s), "
+                f"local {len(local)} file(s), fetch {len(missing)}"
+            )
+        if missing:
+            try:
+                eng.fetch(manifest, staged, include=missing, progress=progress)
+            except EngineError as exc:
+                _raise_integrity_error_if_staged_file_failed(manifest, staged, trust, sel_set, exc)
+                raise
+
+        verdict = verify_manifest(manifest, staged, trust, only=sel_set)
+
+        if not verdict.integrity_ok:
+            bad = [f"{f.path} ({f.reason})" for f in verdict.files if not f.ok]
+            raise IntegrityError("integrity check failed: " + "; ".join(bad))
+        if require_signature and not verdict.origin_ok:
+            raise UnverifiedOriginError(
+                "publisher signature could not be verified: " + "; ".join(verdict.messages)
+            )
+
+        target = place_into_cache(staged, cache_dir, repo_id, rev, manifest=manifest)
+
+    return PullResult(target, repo_id, rev, manifest, verdict)
+
+
+def include_to_paths(manifest: dict, include: Optional[List[str]]) -> List[str]:
+    import fnmatch
+
+    if not include:
+        return [a["path"] for a in manifest["artifacts"]]
+    return [
+        a["path"]
+        for a in manifest["artifacts"]
+        if any(fnmatch.fnmatch(a["path"], pat) for pat in include)
+    ]
+
+
+def _copy_from_local_source(source_dir: Path, staged_dir, paths: List[str]) -> List[str]:
+    source = Path(source_dir).resolve()
+    staged = Path(staged_dir)
+    copied: List[str] = []
+    for artifact_path in paths:
+        rel = _safe_artifact_relpath(artifact_path)
+        src = (source / rel).resolve()
+        try:
+            src.relative_to(source)
+        except ValueError as exc:
+            raise IntegrityError(f"artifact path escapes local source: {artifact_path!r}") from exc
+        if not src.is_file():
+            continue
+        dst = staged / rel
+        dst.parent.mkdir(parents=True, exist_ok=True)
+        shutil.copy2(src, dst)
+        copied.append(artifact_path)
+    return copied
+
+
+def _safe_artifact_relpath(path: str) -> Path:
+    if not isinstance(path, str) or not path:
+        raise IntegrityError(f"unsafe artifact path: {path!r}")
+    rel = PurePosixPath(path)
+    if rel.is_absolute() or not rel.parts or any(part in ("", ".", "..") for part in rel.parts):
+        raise IntegrityError(f"unsafe artifact path: {path!r}")
+    return Path(*rel.parts)
+
+
+def _raise_integrity_error_if_staged_file_failed(
+    manifest: dict,
+    staged_dir,
+    trust: TrustStore,
+    only: set,
+    cause: EngineError,
+) -> None:
+    verdict = verify_manifest(manifest, staged_dir, trust, only=only)
+    bad = [f"{f.path} ({f.reason})" for f in verdict.files if not f.ok and f.reason != "missing"]
+    if bad:
+        raise IntegrityError("integrity check failed: " + "; ".join(bad)) from cause
+
+
+# --- huggingface_hub-compatible surface -------------------------------------
+
+def snapshot_download(repo_id: str, *, revision: Optional[str] = None,
+                      allow_patterns: Optional[List[str]] = None,
+                      cache_dir=None, catalog: Optional[str] = None,
+                      **_ignored) -> str:
+    """Drop-in for huggingface_hub.snapshot_download. Returns the local path."""
+    ref = repo_id if not revision else f"{repo_id}@{revision}"
+    res = pull(ref, catalog=catalog, cache_dir=cache_dir, include=allow_patterns)
+    return str(res.path)
+
+
+def hf_hub_download(repo_id: str, filename: str, *, revision: Optional[str] = None,
+                    cache_dir=None, catalog: Optional[str] = None, **_ignored) -> str:
+    """Drop-in for huggingface_hub.hf_hub_download. Returns the local file path."""
+    ref = repo_id if not revision else f"{repo_id}@{revision}"
+    res = pull(ref, catalog=catalog, cache_dir=cache_dir, include=[filename])
+    return str(res.path / filename)

ossllms-0.1.0/ossllms/cache.py

@@ -0,0 +1,102 @@
+"""HF-compatible cache with a blob store + hardlink dedup.
+
+Layout (identical to huggingface_hub, so transformers/vllm/llama.cpp load unchanged
+AND versions/quants/repos share bytes):
+
+    <cache>/models--<org>--<name>/
+      blobs/<sha256>                 one physical copy per unique file
+      snapshots/<revision>/<path>    hardlink into blobs/
+      refs/<channel>                 e.g. refs/main -> a revision
+
+Identical files across versions point at the same blob inode, so disk cost is the
+size of the UNIQUE bytes, not N x per version. `reuse_from_blobs` lets a pull skip
+downloading any file already present as a blob (the "only changed bytes move"
+update path in docs/VERSIONING.md).
+"""
+from __future__ import annotations
+
+import os
+import shutil
+from pathlib import Path
+from typing import List
+
+
+def repo_folder_name(repo_id: str) -> str:
+    return "models--" + repo_id.replace("/", "--")
+
+
+def repo_dir(cache_root, repo_id: str) -> Path:
+    return Path(cache_root) / repo_folder_name(repo_id)
+
+
+def blobs_dir(cache_root, repo_id: str) -> Path:
+    return repo_dir(cache_root, repo_id) / "blobs"
+
+
+def blob_path(cache_root, repo_id: str, sha256: str) -> Path:
+    return blobs_dir(cache_root, repo_id) / sha256
+
+
+def snapshot_dir(cache_root, repo_id: str, revision: str) -> Path:
+    return repo_dir(cache_root, repo_id) / "snapshots" / revision
+
+
+def has_blob(cache_root, repo_id: str, sha256: str) -> bool:
+    return blob_path(cache_root, repo_id, sha256).exists()
+
+
+def _link_or_copy(src: Path, dst: Path) -> None:
+    dst.parent.mkdir(parents=True, exist_ok=True)
+    if dst.exists():
+        return
+    try:
+        os.link(src, dst)           # hardlink (same filesystem)
+    except OSError:
+        shutil.copy2(src, dst)      # cross-filesystem fallback
+
+
+def reuse_from_blobs(staged_dir, cache_root, repo_id: str, artifacts) -> List[str]:
+    """Hardlink already-present blobs into the staging dir; return reused paths.
+
+    Call before downloading so files already on disk (from a prior version, a quant,
+    or another pull) are not fetched again.
+    """
+    reused: List[str] = []
+    staged = Path(staged_dir)
+    for a in artifacts:
+        bp = blob_path(cache_root, repo_id, a["sha256"])
+        if bp.exists():
+            _link_or_copy(bp, staged / a["path"])
+            reused.append(a["path"])
+    return reused
+
+
+def place_into_cache(staged_dir, cache_root, repo_id: str, revision: str, manifest=None) -> Path:
+    """Store staged files as blobs (keyed by sha256) and hardlink them into the
+    snapshot dir. Files in the manifest are deduped via the blob store; any extras
+    are copied as-is."""
+    target = snapshot_dir(cache_root, repo_id, revision)
+    target.mkdir(parents=True, exist_ok=True)
+    staged = Path(staged_dir)
+    sha_by_path = {a["path"]: a["sha256"] for a in (manifest or {}).get("artifacts", [])}
+
+    for src in staged.rglob("*"):
+        if not src.is_file():
+            continue
+        rel = src.relative_to(staged)
+        sha = sha_by_path.get(str(rel))
+        snap_dst = target / rel
+        snap_dst.parent.mkdir(parents=True, exist_ok=True)
+        if sha:
+            bp = blob_path(cache_root, repo_id, sha)
+            _link_or_copy(src, bp)          # store blob (idempotent)
+            if snap_dst.exists():
+                snap_dst.unlink()
+            _link_or_copy(bp, snap_dst)     # snapshot hardlinks to blob
+        elif not snap_dst.exists():
+            shutil.copy2(src, snap_dst)
+
+    refs = repo_dir(cache_root, repo_id) / "refs"
+    refs.mkdir(parents=True, exist_ok=True)
+    (refs / "main").write_text(revision, encoding="utf-8")
+    return target