hugpy-storage 0.2.2__tar.gz

hugpy_storage-0.2.2/LICENSE

@@ -0,0 +1,41 @@
+hugpy — Source-Available License
+
+Copyright (c) 2026 putkoff (hugpy.ai). All rights reserved.
+
+Permission is granted, free of charge, to use this software ("hugpy") for
+personal and non-commercial purposes, and for time-limited commercial
+evaluation, subject to the following conditions:
+
+1. Non-commercial use means use by an individual for personal purposes, or
+   use by a non-profit or educational institution for its own internal
+   purposes. Any use by, for, or on behalf of a for-profit business or in
+   connection with revenue-generating activity is commercial use — including
+   internal business use, use in producing goods or services, and use on
+   paid engagements.
+
+2. Commercial use requires a commercial license from the copyright holder.
+   Exception: a business may evaluate the software internally for up to
+   thirty (30) days free of charge; continued use after that requires a
+   commercial license.
+
+3. Redistribution of this software, in source or binary form, modified or
+   unmodified, is not permitted without prior written permission from the
+   copyright holder. Downloading the software from an official distribution
+   channel (PyPI, npm, hugpy.ai) is not redistribution.
+
+4. Modification for personal use or internal evaluation is permitted;
+   distribution of modified versions is not.
+
+5. This notice must be retained in all copies or substantial portions of
+   the software.
+
+6. Any use outside these terms automatically terminates this license.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+IN THE SOFTWARE.
+
+For commercial licensing or redistribution permission: https://hugpy.ai

hugpy_storage-0.2.2/PKG-INFO

@@ -0,0 +1,41 @@
+Metadata-Version: 2.4
+Name: hugpy-storage
+Version: 0.2.2
+Summary: Hugpy storage: model download queue and daemon, Hugging Face transport and token, resumable transfers, model sync, physical inventory and on-disk model layout
+Author-email: putkoff <support@hugpy.ai>
+License-Expression: LicenseRef-Proprietary
+Project-URL: Homepage, https://hugpy.ai
+Project-URL: Documentation, https://github.com/hugpy/hugpy/blob/main/py/storage/hugpy_storage/README.md
+Project-URL: Repository, https://github.com/hugpy/hugpy
+Project-URL: Source, https://github.com/hugpy/hugpy/tree/main/py/storage/hugpy_storage
+Project-URL: Issues, https://github.com/hugpy/hugpy/issues
+Project-URL: Changelog, https://github.com/hugpy/hugpy/releases
+Project-URL: Architecture, https://github.com/hugpy/hugpy/blob/main/PARTITION.md
+Keywords: hugpy,llm,self-hosted,huggingface,downloads,model-storage,gguf
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: hugpy-platform
+Requires-Dist: hugpy-control
+Requires-Dist: abstract_essentials
+Requires-Dist: huggingface_hub>=0.20
+Requires-Dist: requests>=2.28
+Requires-Dist: pydantic>=2
+Provides-Extra: test
+Requires-Dist: pytest>=8; extra == "test"
+Requires-Dist: pytest-timeout; extra == "test"
+Dynamic: license-file
+
+# hugpy-storage
+
+`hugpy_storage` — extracted from `abstract_hugpy_dev` as part of the Hugpy
+partition. Ownership and allowed dependencies are declared in
+`py/partition.toml`; see `PARTITION.md` at the workspace root.
+
+Allowed Python dependencies inside the ecosystem: hugpy_platform, hugpy_control.

hugpy_storage-0.2.2/README.md

@@ -0,0 +1,7 @@
+# hugpy-storage
+
+`hugpy_storage` — extracted from `abstract_hugpy_dev` as part of the Hugpy
+partition. Ownership and allowed dependencies are declared in
+`py/partition.toml`; see `PARTITION.md` at the workspace root.
+
+Allowed Python dependencies inside the ecosystem: hugpy_platform, hugpy_control.

hugpy_storage-0.2.2/docs/REGISTRY-DB-INDEX.md

@@ -0,0 +1,109 @@
+# Registry → rebuildable DB index (operator decision, 2026-09-10)
+
+**Decision (operator-approved):** keep `hugpy.json` markers as the per-model
+source of truth co-located with the weights; replace the aggregate JSON
+registries (`projects/model_discovery.json`, `projects/model_manifest.json`)
+with a **database index that is rebuilt from markers** — droppable and
+regenerable at any time, never a second truth.
+
+## Why
+
+The 2026-09-10 incident chain showed the JSON registries' failure modes:
+
+- **Phantom persistence.** The walk registered org dirs (`gguf/<org>`),
+  downloader staging (`<repo>.tmp-<pid>`), `_archive/` and `incoming/` rows;
+  the merge ("re-adopt prior location — don't throw it away") and the shrink
+  guard then kept them alive after the walk bugs were fixed. Rows had to be
+  removed by hand-editing JSON.
+- **Count wobble.** The console model total moved with every download
+  start/finish because staging dirs entered and left the walk.
+- **Swallowed models.** `gguf/ponpoke` registered as a model and pruned the
+  walk, hiding `flux2-klein-9b-uncensored-text-encoder` from the catalog
+  entirely (fixed in `paths.py::is_model_dir` — org-level guard).
+- **Whole-file rewrites** race between central, the worker agent and the
+  downloader; there is no row-level write.
+
+## Shape
+
+- **Markers (`hugpy.json`)** — identity, capability flags, and now the
+  **`quants` manifest** (`[{file, quant, bytes, shards}]`, mmproj excluded,
+  shard sets collapsed), stamped at `write_hugpy_marker` time and kept true by
+  `sync_marker_quants` on every discovery walk (2026-09-10). Markers travel
+  with the files, survive central loss, and make the store self-describing.
+- **DB index** — SQLite beside the comms mirror
+  (`llm_storage/comms/`, same pattern the download queue already uses) or the
+  toolserver Postgres. Tables ≈ models(hub_id, name, framework, tasks,
+  folder, bytes, status, marker_json, walked_at), quants(model, file, quant,
+  bytes, shards). `rebuild()` = walk markers → truncate + reinsert. All
+  readers (`get_models_dict`, /api/models, quant choices, storage proposals)
+  move to the index; the JSONs become export artifacts or die.
+- **No shrink guard needed**: a degraded-mount walk is detected the same way,
+  but the remedy is "don't rebuild now", not "never delete rows".
+
+## Prerequisite already done
+
+Discovery hardening shipped 2026-09-10 (src + both installed wheels):
+org-level guard in `is_model_dir`, `.tmp-` exclusion, `_archive`/`incoming`/
+`dedupes` excluded from the walk, quant manifest on markers. A future
+`backfill_markers` sweep (exists in `hugpy_marker.py`) will stamp the last
+unstamped dirs so the index can be built from markers alone.
+
+## Phase 1 — SHIPPED 2026-09-10
+
+- Database `hugpy` (owner: user `hugpy`) on the toolserver's Postgres server —
+  same infra, separate database, toolserver's tables untouched.
+- `imports/src/model_index.py`: env-gated (`HUGPY_REGISTRY_DB=pg` +
+  `HUGPY_REGISTRY_PG_DSN`, URI form — a space-separated DSN breaks shell
+  sourcing of the env file), autocommit + `conn.transaction()` (psycopg3's
+  `with conn:` CLOSES the connection — do not reintroduce it).
+- Dual-write: `discover_models` saves the report to the DB after the shrink
+  guard (a refused degraded walk never reaches the DB; an empty report never
+  truncates). JSON remains the fallback/export.
+- DB-first read: `models_config._load_discovery_report` — DB when enabled and
+  populated, else the JSON path byte-identical to before. Workers (no flag,
+  no psycopg) are untouched.
+- `model_quants` fills from marker `quants` riding the report rows; it grows
+  as markers get stamped/synced (backfill_markers sweep = the remaining step
+  to full coverage).
+
+## Next phases (operator: "most everything for hugpy should be converted to db")
+
+1. `model_manifest.json` (download registry) → table, same dual-write pattern.
+2. Marker backfill sweep so model_quants covers the whole store.
+3. Runtime settings / overrides / priority groups stores → tables.
+4. Retire the JSON files once each reader is DB-first and burned in.
+
+## Metrics card — SHIPPED 2026-09-10 (same session)
+
+Modeled on the operator's `llm_storage/assets/metrics.xlsx_0.ods`:
+- `model_metrics` — rollup card, one row per (model × quant × alloc_mode ×
+  worker): cold/hot load s (fed later by load probes/battery), last +
+  running-average tok/s, n_samples, upload_time_s, media_bytes, task.
+- `model_calls` — append-only per-call ledger (every serve: tok/s,
+  prompt/completion tokens, elapsed, request_id, config-state snapshot).
+- Recorder hook: `workers.record_serve_metrics` (the one tok/s writer) —
+  captures worker/alloc/moe/4-bit inside the store lock, writes outside it.
+- Read: `GET /api/llm/models/<key>/metrics` → {metrics, worker_averages,
+  calls}; alias-tolerant (`Jackrong~X` ≡ `X`).
+- Seeded from the fleet's live model_tok_stats (31 rows).
+- Gotcha fixed: PID fork-guard on the PG connection (gunicorn forks after a
+  boot-time connect; a shared socket = "SSL bad record mac").
+
+## DB code layout — STANDARD (operator ruling, 2026-09-10)
+
+All hugpy DB creation and calls follow the explicit repository pattern of
+solcatcher's `src/db/repositories/repos` — implemented for the model index at
+`imports/src/model_index/` and REQUIRED for every future table:
+
+    query_registry.py   EVERY SQL statement, named constants grouped per
+                        repository (CREATE_TABLE + CREATE_INDEXES first,
+                        then the named operations). Nothing inlines SQL.
+    client.py           DatabaseClient — DSN resolution, PID fork-guard,
+                        psycopg3 autocommit + transaction(), fail-open
+                        mark_unavailable(). The only file that imports psycopg.
+    repositories.py     One class per table, one explicit method per query.
+                        Thin: parameters in, rows out.
+    service.py          The business layer — enabled() gate, locking,
+                        transactions, alias forms, guards, degradation.
+    __init__.py         Explicit exports + a module-level singleton exposing
+                        the plain-function API call sites use.

hugpy_storage-0.2.2/pyproject.toml

@@ -0,0 +1,71 @@
+[build-system]
+requires = ["setuptools>=77", "setuptools-scm>=8"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "hugpy-storage"
+dynamic = ["version"]
+description = "Hugpy storage: model download queue and daemon, Hugging Face transport and token, resumable transfers, model sync, physical inventory and on-disk model layout"
+readme = "README.md"
+requires-python = ">=3.10"
+license = "LicenseRef-Proprietary"
+license-files = ["LICENSE"]
+authors = [{ name = "putkoff", email = "support@hugpy.ai" }]
+keywords = [
+    "hugpy",
+    "llm",
+    "self-hosted",
+    "huggingface",
+    "downloads",
+    "model-storage",
+    "gguf",
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = [
+    "hugpy-platform",
+    "hugpy-control",
+    "abstract_essentials",
+    "huggingface_hub>=0.20",
+    "requests>=2.28",
+    "pydantic>=2",
+]
+
+[project.urls]
+Homepage = "https://hugpy.ai"
+Documentation = "https://github.com/hugpy/hugpy/blob/main/py/storage/hugpy_storage/README.md"
+Repository = "https://github.com/hugpy/hugpy"
+Source = "https://github.com/hugpy/hugpy/tree/main/py/storage/hugpy_storage"
+Issues = "https://github.com/hugpy/hugpy/issues"
+Changelog = "https://github.com/hugpy/hugpy/releases"
+Architecture = "https://github.com/hugpy/hugpy/blob/main/PARTITION.md"
+
+[project.optional-dependencies]
+test = ["pytest>=8", "pytest-timeout"]
+
+[project.scripts]
+hugpy-storage = "hugpy_storage.cli:main"
+# Transitional alias for the systemd unit / operators (PARTITION.md).
+hugpy-downloader = "hugpy_storage.downloader.daemon:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+hugpy_storage = ["py.typed", "**/*.json", "**/*.sh", "**/*.md", "**/*.txt", "deploy/*.service"]
+
+# ---------------------------------------------------------------------------
+# Lockstep workspace version (2026-09-22): every in-tree hugpy-* distribution
+# takes ONE version from the workspace git tag (vX.Y.Z at the repo root), so a
+# release is a tag and a build always carries the commit it came from
+# (X.Y.Z.devN+g<sha>[.dirty] between tags). A checkout without git metadata
+# builds as 0.0.0+unknown, which central refuses to advertise to workers.
+[tool.setuptools_scm]
+root = "../../.."
+fallback_version = "0.0.0+unknown"

hugpy_storage-0.2.2/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

hugpy_storage-0.2.2/src/hugpy_storage/__init__.py

@@ -0,0 +1,56 @@
+"""hugpy_storage — bytes at rest and in transit.
+
+The download queue and daemon (``downloader``), resumable central/HF transfer
+(``provision``, ``model_sync``, ``download_models``), Hugging Face transport
+and token (``huggingface_api``, ``hf_token``, ``model_metadata``), physical
+inventory and its caches (``model_physical``, ``model_status_cache``,
+``model_presence``), the on-disk model layout (``model_paths``,
+``hugpy_marker``, ``gguf_inspect``) and the console-side helpers.
+
+Seams to the layers above (installed by the composition root):
+
+* :mod:`hugpy_storage.catalog_source` — the model registry Protocol
+  (``set_catalog_source``); null default knows no models.
+* :mod:`hugpy_storage.providers` — budget gate, transfer telemetry, executor
+  registrar, serve-path hook, footprint selector.
+* :mod:`hugpy_storage.events` — ``catalog.changed`` on the control bus after
+  the inventory moves.
+
+This ``__init__`` is deliberately light: nothing heavy (huggingface_hub,
+requests, pydantic, sqlite stores) is imported here.
+"""
+from __future__ import annotations
+
+from hugpy_storage.catalog_source import (
+    CatalogSource,
+    DictCatalogSource,
+    NullCatalogSource,
+    get_catalog_source,
+    set_catalog_source,
+)
+from hugpy_storage.events import publish_catalog_changed
+from hugpy_storage.providers import (
+    set_budget_gate,
+    set_executor_registrar,
+    set_footprint_selector,
+    set_serve_path_hook,
+    set_transfer_telemetry,
+)
+
+try:  # the installed distribution's version: the workspace tag/commit, never a literal
+    from importlib.metadata import version as _dist_version
+    __version__ = _dist_version("hugpy-storage")
+except Exception:  # noqa: BLE001 — source tree without metadata
+    __version__ = "0.0.0+unknown"
+
+__all__ = [
+    "__version__",
+    # catalog seam
+    "CatalogSource", "NullCatalogSource", "DictCatalogSource",
+    "set_catalog_source", "get_catalog_source",
+    # upper-layer hooks
+    "set_budget_gate", "set_transfer_telemetry", "set_executor_registrar",
+    "set_serve_path_hook", "set_footprint_selector",
+    # events
+    "publish_catalog_changed",
+]