withcache 0.10.0__tar.gz → 0.12.0__tar.gz

{withcache-0.10.0 → withcache-0.12.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: withcache
-Version: 0.10.0
+Version: 0.12.0
 Summary: Operator-curated, URL-keyed artifact cache for a small lab (CUDA/ROCm/DOCA/firmware)
 Project-URL: Homepage, https://github.com/safl/withcache
 Author-email: "Simon A. F. Lund" <safl@safl.dk>
@@ -46,21 +46,23 @@ Artifacts are cached **by their origin URL as a key**; the shim opts in by
 re-pointing the URL at the cache. No transparent proxy, no TLS interception, no
 client CA. The URL is a lookup key, not a connection target.
 
-By default a miss is **auto-fetched**: the request falls through to origin (so
-the caller gets its file straight away), and the cache-host pulls the same
-artifact in the background, so the next request hits. Run with **`--curate`** to
-require a human instead, who reviews the miss list in a small web UI and presses
-**Download** (or pre-seeds via the Downloads-page **Fetch** form). Either way the cache-host is the
-only box that needs internet egress (and any vendor credentials), and clients
-never write to it.
+A miss falls through to origin (the caller gets its file straight away) and
+withcache records the miss on the operator's **`/ui/misses`** page. The operator
+reviews the miss list, picks what's worth caching, and one click **Fetch**s a URL
+into the catalog + downloads it. Every entry in the catalog is a byte-perfect
+copy of the origin at the time the operator hit Download; subsequent requests
+hit the local cache. The cache-host is the only box that needs internet egress
+(and any vendor credentials); clients never write to it.
 
 ## Why not just curl + a caching proxy?
 
 For `https://` (i.e. every vendor download) a forward proxy can't cache without
 **SSL-bump / MITM**: curl tunnels TLS end-to-end via `CONNECT`, so the proxy
 only sees ciphertext. The shim sidesteps that entirely by *re-pointing the URL*
-to the cache instead of intercepting the connection. And no proxy offers the
-optional **operator-curated** model (`--curate`: a miss queue a human approves).
+to the cache instead of intercepting the connection. And a proxy that
+auto-fetches everything a client asks for isn't what you want in a lab; the
+**operator-curated** model here means only bytes the operator chose live on
+disk.
 
 ## Components
 
@@ -117,9 +119,8 @@ WITHCACHE_ADMIN_PASSWORD=change-me withcache-server --data-dir ./data --port 808
 Data (blobs + `cache.db` + `session-secret`) lives in the `/data` volume (or
 `--data-dir`). Artifacts are immutable per version, so there's no cache
 invalidation. `--workers N` sets the number of concurrent download workers,
-`--curate` switches from auto-fetch to operator-approved pulls, and `--max-bytes`
-(e.g. `50G`) caps the cache: when full it refuses new fills (no auto-eviction),
-and you free space by deleting artifacts in the UI.
+`--max-bytes` (e.g. `50G`) caps the cache: when full it refuses new fills (no
+auto-eviction), and you free space by deleting artifacts in the UI.
 
 ## Use the shims (transparent `curl` / `wget`)
 
@@ -245,10 +246,10 @@ Notes & limits (all degrade gracefully; worst case is "no caching, curl still wo
 `http://withcache-server:8081/` (Bootstrap 5 + Bootstrap Icons + HTMX, bundled
 offline; matches bty's chrome for a consistent trio) is a five-page dashboard:
 - **Cached** (landing): URL, size, **hits** (times served) and **misses** (times requested before it was cached), SHA-256, fetched-at, each with **Delete** to free space.
-- **Streams**: in-flight stream-through-and-store fetches serving bytes to a client while writing to disk.
-- **Downloads**: live progress bars, `queued/running/completed/cancelled/failed`, **Cancel**, and **Clear finished**. Downloads run in a background worker pool, not in the request, so large pulls never block, modelled on [bty]'s job managers. The subnav Fetch form pre-seeds an artifact before anyone misses it.
-- **Misses**: auto-fetched by default, or (under `--curate`) each with **Download** (queues a background pull) and **Dismiss**.
-- **Catalog**: image catalog fetched from a nosi-style `catalog.toml` (URL configurable via `$WITHCACHE_CATALOG_URL` or the subnav Set&fetch input); pre-seed by URL via the "Add image from oras" input.
+- **Downloads**: live progress bars, `queued/running/completed/cancelled/failed`, **Cancel**, and **Clear finished**. Downloads run in a background worker pool, not in the request, so large pulls never block, modelled on [bty]'s job managers.
+- **Misses**: URLs clients asked for that aren't downloaded yet. Each with **Fetch** (promotes to a catalog entry AND downloads it) and **Dismiss** (forget it).
+- **Catalog**: image catalog fetched from a nosi-style `catalog.toml` (URL configurable via `$WITHCACHE_CATALOG_URL` or the subnav Set&fetch input); pre-seed by URL via the "Add image from oras" or "Add HTTPS entry" forms, then **Download** each entry.
+- **Settings**: identity + storage paths + catalog source + logging (uvicorn level) + auth.
 
 ## Auth
 
@@ -288,10 +289,10 @@ url = client.serve_url("http://cache:8081", origin) or origin
 ```
 
 `is_cached()` is a graceful `HEAD` (a miss, timeout, or unreachable cache all
-return `False`, so you fall back to the origin), and it doubles as a warm-up:
-the probe records the miss and, in auto-fetch mode, enqueues the fill, so the
-next call flips to the cache. The encoding is shared with the shims and server,
-so consumers stay in lockstep with the cache-host.
+return `False`, so you fall back to the origin). Since v0.10.0 a miss is
+recorded on the cache-host's `/ui/misses` page but no background fetch fires
+-- the operator explicitly chooses what to Download. The encoding is shared
+with the shims and server, so consumers stay in lockstep with the cache-host.
 
 ### Pull an `oras://` artifact (oras + client together)
 

{withcache-0.10.0 → withcache-0.12.0}/README.md

@@ -23,21 +23,23 @@ Artifacts are cached **by their origin URL as a key**; the shim opts in by
 re-pointing the URL at the cache. No transparent proxy, no TLS interception, no
 client CA. The URL is a lookup key, not a connection target.
 
-By default a miss is **auto-fetched**: the request falls through to origin (so
-the caller gets its file straight away), and the cache-host pulls the same
-artifact in the background, so the next request hits. Run with **`--curate`** to
-require a human instead, who reviews the miss list in a small web UI and presses
-**Download** (or pre-seeds via the Downloads-page **Fetch** form). Either way the cache-host is the
-only box that needs internet egress (and any vendor credentials), and clients
-never write to it.
+A miss falls through to origin (the caller gets its file straight away) and
+withcache records the miss on the operator's **`/ui/misses`** page. The operator
+reviews the miss list, picks what's worth caching, and one click **Fetch**s a URL
+into the catalog + downloads it. Every entry in the catalog is a byte-perfect
+copy of the origin at the time the operator hit Download; subsequent requests
+hit the local cache. The cache-host is the only box that needs internet egress
+(and any vendor credentials); clients never write to it.
 
 ## Why not just curl + a caching proxy?
 
 For `https://` (i.e. every vendor download) a forward proxy can't cache without
 **SSL-bump / MITM**: curl tunnels TLS end-to-end via `CONNECT`, so the proxy
 only sees ciphertext. The shim sidesteps that entirely by *re-pointing the URL*
-to the cache instead of intercepting the connection. And no proxy offers the
-optional **operator-curated** model (`--curate`: a miss queue a human approves).
+to the cache instead of intercepting the connection. And a proxy that
+auto-fetches everything a client asks for isn't what you want in a lab; the
+**operator-curated** model here means only bytes the operator chose live on
+disk.
 
 ## Components
 
@@ -94,9 +96,8 @@ WITHCACHE_ADMIN_PASSWORD=change-me withcache-server --data-dir ./data --port 808
 Data (blobs + `cache.db` + `session-secret`) lives in the `/data` volume (or
 `--data-dir`). Artifacts are immutable per version, so there's no cache
 invalidation. `--workers N` sets the number of concurrent download workers,
-`--curate` switches from auto-fetch to operator-approved pulls, and `--max-bytes`
-(e.g. `50G`) caps the cache: when full it refuses new fills (no auto-eviction),
-and you free space by deleting artifacts in the UI.
+`--max-bytes` (e.g. `50G`) caps the cache: when full it refuses new fills (no
+auto-eviction), and you free space by deleting artifacts in the UI.
 
 ## Use the shims (transparent `curl` / `wget`)
 
@@ -222,10 +223,10 @@ Notes & limits (all degrade gracefully; worst case is "no caching, curl still wo
 `http://withcache-server:8081/` (Bootstrap 5 + Bootstrap Icons + HTMX, bundled
 offline; matches bty's chrome for a consistent trio) is a five-page dashboard:
 - **Cached** (landing): URL, size, **hits** (times served) and **misses** (times requested before it was cached), SHA-256, fetched-at, each with **Delete** to free space.
-- **Streams**: in-flight stream-through-and-store fetches serving bytes to a client while writing to disk.
-- **Downloads**: live progress bars, `queued/running/completed/cancelled/failed`, **Cancel**, and **Clear finished**. Downloads run in a background worker pool, not in the request, so large pulls never block, modelled on [bty]'s job managers. The subnav Fetch form pre-seeds an artifact before anyone misses it.
-- **Misses**: auto-fetched by default, or (under `--curate`) each with **Download** (queues a background pull) and **Dismiss**.
-- **Catalog**: image catalog fetched from a nosi-style `catalog.toml` (URL configurable via `$WITHCACHE_CATALOG_URL` or the subnav Set&fetch input); pre-seed by URL via the "Add image from oras" input.
+- **Downloads**: live progress bars, `queued/running/completed/cancelled/failed`, **Cancel**, and **Clear finished**. Downloads run in a background worker pool, not in the request, so large pulls never block, modelled on [bty]'s job managers.
+- **Misses**: URLs clients asked for that aren't downloaded yet. Each with **Fetch** (promotes to a catalog entry AND downloads it) and **Dismiss** (forget it).
+- **Catalog**: image catalog fetched from a nosi-style `catalog.toml` (URL configurable via `$WITHCACHE_CATALOG_URL` or the subnav Set&fetch input); pre-seed by URL via the "Add image from oras" or "Add HTTPS entry" forms, then **Download** each entry.
+- **Settings**: identity + storage paths + catalog source + logging (uvicorn level) + auth.
 
 ## Auth
 
@@ -265,10 +266,10 @@ url = client.serve_url("http://cache:8081", origin) or origin
 ```
 
 `is_cached()` is a graceful `HEAD` (a miss, timeout, or unreachable cache all
-return `False`, so you fall back to the origin), and it doubles as a warm-up:
-the probe records the miss and, in auto-fetch mode, enqueues the fill, so the
-next call flips to the cache. The encoding is shared with the shims and server,
-so consumers stay in lockstep with the cache-host.
+return `False`, so you fall back to the origin). Since v0.10.0 a miss is
+recorded on the cache-host's `/ui/misses` page but no background fetch fires
+-- the operator explicitly chooses what to Download. The encoding is shared
+with the shims and server, so consumers stay in lockstep with the cache-host.
 
 ### Pull an `oras://` artifact (oras + client together)
 

{withcache-0.10.0 → withcache-0.12.0}/shim/build.zig.zon

@@ -2,7 +2,7 @@
     .name = .withcache_shim,
     // Zig requires a literal here; keep it in lockstep with the project's
     // single source (src/withcache/__init__.py) via `make bump` / `make version-check`.
-    .version = "0.10.0",
+    .version = "0.12.0",
     .fingerprint = 0xd7d96c5ed212ccaa,
     .minimum_zig_version = "0.16.0",
     .paths = .{

{withcache-0.10.0 → withcache-0.12.0}/src/withcache/__init__.py

@@ -7,9 +7,10 @@
   a cache-host (build serve URLs, probe what's cached) without re-implementing
   the ``/b/`` URL scheme.
 - ``withcache.oras``: OCI registry adapter. Parses ``oras://...`` references
-  and resolves them to a plain HTTPS blob URL + bearer headers. The cache-host
-  uses it on a cold miss; library consumers (e.g. ``bty``) import it to
-  validate catalog entries and pre-resolve content digests.
+  and resolves them to a plain HTTPS blob URL + bearer headers. The
+  cache-host's DownloadManager invokes it when an operator hits Download
+  on an ``oras://`` catalog entry; library consumers (e.g. ``bty``) import
+  it to validate catalog entries and pre-resolve content digests.
 
 Since v0.9.0 the daemon runs on FastAPI + Jinja + Bootstrap 5 + htmx
 (matching bty-web + nbdmux, the eventual ``trio-common`` extraction
@@ -21,6 +22,6 @@ inherit the framework floor.
 from . import oras
 from .client import blob_url, cache_base, is_cached, serve_url
 
-__version__ = "0.10.0"
+__version__ = "0.12.0"
 
 __all__ = ["__version__", "blob_url", "cache_base", "is_cached", "oras", "serve_url"]

{withcache-0.10.0 → withcache-0.12.0}/src/withcache/_api.py

@@ -11,15 +11,15 @@ stay byte-identical:
   decorative; the b64 token is the source of truth.
 - Cache hit -> 200 + ``Content-Type`` + ``Content-Length`` +
   ``X-Withcache-Sha256`` + streamed body (64 KiB chunks).
-- Cache miss -> 404 with ``cache miss (recorded)`` body,
-  side-effect of enqueueing a background fetch when
-  ``auto_fetch`` is on and the store has capacity.
-- HEAD support: same headers, no body (probes from
-  bty.web._withcache.is_cached use HEAD).
-- ``Authorization`` request header forwarded onto the background
-  fetch worker so a token-gated origin (typical use case: an OCI
-  bearer minted by bty at catalog import time) can be pulled by
-  the worker with the same credential.
+- Cache miss -> 404 with a message pointing at ``/ui/catalog``'s
+  Download button, plus a side-effect ``record_miss`` write so the
+  ``/ui/misses`` page shows the operator what clients tried to
+  fetch that isn't in the catalog yet.
+
+Since v0.10.0 there is no auto-fetch on miss: the operator
+explicitly Downloads. The Miss page's Fetch button (retargeted in
+v0.11.0) promotes a missed URL to a catalog entry + enqueues the
+Download in one click.
 
 Streaming shape: FastAPI ``StreamingResponse`` with a generator
 that reads chunks from the on-disk blob at 64 KiB reads;
@@ -37,7 +37,7 @@ from typing import Any
 from fastapi import Depends, FastAPI, HTTPException, Request
 from fastapi.responses import JSONResponse, PlainTextResponse, Response, StreamingResponse
 
-from .server import CHUNK, CatalogState, StreamRegistry, _oras_tag_moved, _serialise_catalog
+from .server import CHUNK, CatalogState, _oras_tag_moved, _serialise_catalog
 
 
 def _decode_blob_origin(path: str, query: str) -> str:
@@ -78,7 +78,6 @@ def _serve_blob(
         return PlainTextResponse("missing url\n", status_code=400)
 
     store = request.app.state.store
-    streams: StreamRegistry = request.app.state.streams
 
     row = store.get_blob(url)
     if row is not None and _oras_tag_moved(url, row["sha256"]):
@@ -105,35 +104,20 @@ def _serve_blob(
 
     if head_only:
         # No body; the shim's HEAD probe doesn't count as a served
-        # download so record_hit + stream registration are skipped.
+        # download so record_hit is skipped.
         return Response(status_code=200, media_type=media_type, headers=headers)
 
     store.record_hit(row["key"])
-    client = f"{request.client.host}:{request.client.port}" if request.client else "unknown"
-    stream = streams.start(url=url, client=client, total=row["size"])
 
     def _chunks() -> Iterator[bytes]:
         """Read the blob in 64 KiB chunks. StreamingResponse iterates
-        this + writes each chunk to the client. Progress ticks fire
-        every :data:`StreamRegistry.PROGRESS_STRIDE` chunks so the
-        dashboard's 1 Hz refresh sees updates without lock-contention
-        on a busy box."""
-        sent = 0
-        ticks = 0
-        try:
-            with open(path, "rb") as f:
-                while True:
-                    chunk = f.read(CHUNK)
-                    if not chunk:
-                        break
-                    yield chunk
-                    sent += len(chunk)
-                    ticks += 1
-                    if ticks % StreamRegistry.PROGRESS_STRIDE == 0:
-                        streams.bump(stream.id, sent)
-                streams.bump(stream.id, sent)
-        finally:
-            streams.finish(stream.id)
+        this + writes each chunk to the client."""
+        with open(path, "rb") as f:
+            while True:
+                chunk = f.read(CHUNK)
+                if not chunk:
+                    break
+                yield chunk
 
     return StreamingResponse(_chunks(), media_type=media_type, headers=headers)
 
@@ -141,10 +125,9 @@ def _serve_blob(
 def register_api_routes(app: FastAPI) -> None:
     """Attach the byte-serving routes to ``app``.
 
-    Runtime objects (``store``, ``mgr``, ``streams``, ``auto_fetch``)
-    are read from ``app.state`` at request time so tests + the
-    lifespan hook can inject fresh instances without rebuilding
-    the app.
+    Runtime objects (``store``, ``mgr``, ``streams``) are read from
+    ``app.state`` at request time so tests + the lifespan hook can
+    inject fresh instances without rebuilding the app.
     """
 
     @app.get("/blob", response_class=Response)
@@ -212,25 +195,25 @@ def register_api_routes(app: FastAPI) -> None:
 
         Open route: bty polls this from a sibling container without a
         session (mirrors the ``/blob`` byte-serving surface's trust
-        model -- LAN-only, no auth on reads). Returns a snapshot of
-        the in-process ``CatalogState.entries`` plus provenance
-        metadata (URL, env pin, fetch timestamps) so clients can
-        detect staleness. Every entry is enriched with a
-        ``downloaded_at`` field (or ``None``) so nbdmux + bty can
-        filter to only-downloaded entries without a second call. No
-        pagination -- the catalog is small (dozens of entries at
+        model -- LAN-only, no auth on reads). Returns only entries
+        whose bytes are on disk in withcache's cache. Since v0.11.0
+        "presence in this list" IS the readiness signal: staged
+        entries (added but not yet Downloaded) are invisible to trio
+        consumers so bty + nbdmux can trust every entry they see is
+        flashable / exportable. Operators see the staged view on
+        withcache's own ``/ui/catalog``.
+
+        No pagination -- the catalog is small (dozens of entries at
         most).
         """
         cs: CatalogState = request.app.state.catalog
         store = request.app.state.store
         entries_out: list[dict[str, Any]] = []
         for e in cs.entries:
-            enriched = dict(e)
             fetch_url = e.get("resolved_src") or e.get("src") or ""
-            row = store.get_blob(fetch_url) if fetch_url else None
-            enriched["downloaded_at"] = row["fetched_at"] if row else None
-            enriched["downloaded_size"] = int(row["size"]) if row else None
-            entries_out.append(enriched)
+            if not fetch_url or store.get_blob(fetch_url) is None:
+                continue
+            entries_out.append(dict(e))
         return JSONResponse(
             {
                 "url": cs.url,