withcache 0.2.0__tar.gz → 0.4.0__tar.gz

{withcache-0.2.0 → withcache-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: withcache
-Version: 0.2.0
+Version: 0.4.0
 Summary: Operator-curated, URL-keyed artifact cache for a small lab (CUDA/ROCm/DOCA/firmware)
 Project-URL: Homepage, https://github.com/safl/withcache
 Author-email: "Simon A. F. Lund" <safl@safl.dk>
@@ -18,7 +18,7 @@ Description-Content-Type: text/markdown
 
 # withcache
 
-[![ci](https://github.com/safl/withcache/actions/workflows/ci.yml/badge.svg)](https://github.com/safl/withcache/actions/workflows/ci.yml)
+[![ci](https://github.com/safl/withcache/actions/workflows/ci-cd.yml/badge.svg)](https://github.com/safl/withcache/actions/workflows/ci-cd.yml)
 [![PyPI](https://img.shields.io/pypi/v/withcache.svg)](https://pypi.org/project/withcache/)
 [![license](https://img.shields.io/badge/license-BSD--3--Clause-blue.svg)](LICENSE)
 [![built with Zig](https://img.shields.io/badge/built%20with-Zig%200.16.0-f7a41d.svg)](https://ziglang.org)
@@ -111,8 +111,10 @@ WITHCACHE_ADMIN_PASSWORD=change-me withcache-server --data-dir ./data --port 300
 
 Data (blobs + `cache.db` + `session-secret`) lives in the `/data` volume (or
 `--data-dir`). Artifacts are immutable per version, so there's no cache
-invalidation. `--workers N` sets the number of concurrent download workers, and
-`--curate` switches from auto-fetch to operator-approved pulls.
+invalidation. `--workers N` sets the number of concurrent download workers,
+`--curate` switches from auto-fetch to operator-approved pulls, and `--max-bytes`
+(e.g. `50G`) caps the cache: when full it refuses new fills (no auto-eviction),
+and you free space by deleting artifacts in the UI.
 
 ## Use the shims (transparent `curl` / `wget`)
 
@@ -238,7 +240,7 @@ Notes & limits (all degrade gracefully; worst case is "no caching, curl still wo
 `http://withcache-server:3000/` (Pico.css + HTMX, bundled offline) shows:
 - **Misses**: auto-fetched by default, or (under `--curate`) each with **Download** (queues a background pull) and **Dismiss**.
 - **Downloads**: live progress bars, `queued/running/completed/cancelled/failed`, **Cancel**, and **Clear finished**. Downloads run in a background worker pool, not in the request, so large pulls never block, modelled on [bty]'s job managers.
-- **Cached artifacts**: URL, size, **hits** (times served) and **misses** (times requested before it was cached), SHA-256, fetched-at.
+- **Cached artifacts**: URL, size, **hits** (times served) and **misses** (times requested before it was cached), SHA-256, fetched-at, each with **Delete** to free space.
 - **Add from URI**: pre-seed an artifact before anyone misses it.
 
 ## Auth
@@ -264,6 +266,25 @@ CDN/presigned URLs (whose tokens change every request) still match by path. Pass
 (`.deb`/`.rpm`) are GPG-signed and verified by the client regardless of
 transport, so caching them this way is safe.
 
+## Consume from another tool (the client library)
+
+A tool that already knows its download URLs (e.g. an installer or a provisioner)
+can prefer the cache without shelling out to a shim or re-implementing the `/b/`
+scheme. `withcache.client` is stdlib-only, so importing it adds no dependencies:
+
+```python
+from withcache import client
+
+# "use the cache when it's warm, the origin otherwise"
+url = client.serve_url("http://cache:3000", origin) or origin
+```
+
+`is_cached()` is a graceful `HEAD` (a miss, timeout, or unreachable cache all
+return `False`, so you fall back to the origin), and it doubles as a warm-up:
+the probe records the miss and, in auto-fetch mode, enqueues the fill, so the
+next call flips to the cache. The encoding is shared with the shims and server,
+so consumers stay in lockstep with the cache-host.
+
 ## Tests
 
 ```sh

{withcache-0.2.0 → withcache-0.4.0}/README.md

@@ -1,6 +1,6 @@
 # withcache
 
-[![ci](https://github.com/safl/withcache/actions/workflows/ci.yml/badge.svg)](https://github.com/safl/withcache/actions/workflows/ci.yml)
+[![ci](https://github.com/safl/withcache/actions/workflows/ci-cd.yml/badge.svg)](https://github.com/safl/withcache/actions/workflows/ci-cd.yml)
 [![PyPI](https://img.shields.io/pypi/v/withcache.svg)](https://pypi.org/project/withcache/)
 [![license](https://img.shields.io/badge/license-BSD--3--Clause-blue.svg)](LICENSE)
 [![built with Zig](https://img.shields.io/badge/built%20with-Zig%200.16.0-f7a41d.svg)](https://ziglang.org)
@@ -93,8 +93,10 @@ WITHCACHE_ADMIN_PASSWORD=change-me withcache-server --data-dir ./data --port 300
 
 Data (blobs + `cache.db` + `session-secret`) lives in the `/data` volume (or
 `--data-dir`). Artifacts are immutable per version, so there's no cache
-invalidation. `--workers N` sets the number of concurrent download workers, and
-`--curate` switches from auto-fetch to operator-approved pulls.
+invalidation. `--workers N` sets the number of concurrent download workers,
+`--curate` switches from auto-fetch to operator-approved pulls, and `--max-bytes`
+(e.g. `50G`) caps the cache: when full it refuses new fills (no auto-eviction),
+and you free space by deleting artifacts in the UI.
 
 ## Use the shims (transparent `curl` / `wget`)
 
@@ -220,7 +222,7 @@ Notes & limits (all degrade gracefully; worst case is "no caching, curl still wo
 `http://withcache-server:3000/` (Pico.css + HTMX, bundled offline) shows:
 - **Misses**: auto-fetched by default, or (under `--curate`) each with **Download** (queues a background pull) and **Dismiss**.
 - **Downloads**: live progress bars, `queued/running/completed/cancelled/failed`, **Cancel**, and **Clear finished**. Downloads run in a background worker pool, not in the request, so large pulls never block, modelled on [bty]'s job managers.
-- **Cached artifacts**: URL, size, **hits** (times served) and **misses** (times requested before it was cached), SHA-256, fetched-at.
+- **Cached artifacts**: URL, size, **hits** (times served) and **misses** (times requested before it was cached), SHA-256, fetched-at, each with **Delete** to free space.
 - **Add from URI**: pre-seed an artifact before anyone misses it.
 
 ## Auth
@@ -246,6 +248,25 @@ CDN/presigned URLs (whose tokens change every request) still match by path. Pass
 (`.deb`/`.rpm`) are GPG-signed and verified by the client regardless of
 transport, so caching them this way is safe.
 
+## Consume from another tool (the client library)
+
+A tool that already knows its download URLs (e.g. an installer or a provisioner)
+can prefer the cache without shelling out to a shim or re-implementing the `/b/`
+scheme. `withcache.client` is stdlib-only, so importing it adds no dependencies:
+
+```python
+from withcache import client
+
+# "use the cache when it's warm, the origin otherwise"
+url = client.serve_url("http://cache:3000", origin) or origin
+```
+
+`is_cached()` is a graceful `HEAD` (a miss, timeout, or unreachable cache all
+return `False`, so you fall back to the origin), and it doubles as a warm-up:
+the probe records the miss and, in auto-fetch mode, enqueues the fill, so the
+next call flips to the cache. The encoding is shared with the shims and server,
+so consumers stay in lockstep with the cache-host.
+
 ## Tests
 
 ```sh

{withcache-0.2.0 → withcache-0.4.0}/deploy/Containerfile

@@ -4,8 +4,11 @@
 FROM python:3.12-slim
 
 # Install the package (no third-party deps) to get the withcache-server command.
+# hatch_build.py is the wheel build hook (ships the shims); without it the build
+# fails. No zig in this image, so the shims install as Python launchers, which
+# is fine -- the container only runs withcache-server.
 WORKDIR /app
-COPY pyproject.toml README.md /app/
+COPY pyproject.toml README.md hatch_build.py /app/
 COPY src /app/src
 RUN pip install --no-cache-dir /app
 

{withcache-0.2.0 → withcache-0.4.0}/shim/build.zig.zon

@@ -2,7 +2,7 @@
     .name = .withcache_shim,
     // Zig requires a literal here; keep it in lockstep with the project's
     // single source (src/withcache/__init__.py) via `make bump` / `make version-check`.
-    .version = "0.2.0",
+    .version = "0.4.0",
     .fingerprint = 0xd7d96c5ed212ccaa,
     .minimum_zig_version = "0.16.0",
     .paths = .{

withcache-0.4.0/src/withcache/__init__.py

@@ -0,0 +1,17 @@
+"""withcache — operator-curated, URL-keyed artifact cache for a small lab.
+
+- ``withcache-server`` (withcache.server:main): the cache-host.
+- ``curlwithcache`` / ``wgetwithcache``: transparent curl/wget shims, shipped
+  as a native binary or a Python launcher (see hatch_build.py).
+- ``withcache.client``: a tiny, stdlib-only library for other tools to consume
+  a cache-host (build serve URLs, probe what's cached) without re-implementing
+  the ``/b/`` URL scheme.
+
+All modules are stdlib-only and self-contained.
+"""
+
+from .client import blob_url, cache_base, is_cached, serve_url
+
+__version__ = "0.4.0"
+
+__all__ = ["__version__", "blob_url", "cache_base", "is_cached", "serve_url"]

withcache-0.4.0/src/withcache/client.py

@@ -0,0 +1,89 @@
+"""A tiny client for consuming a withcache cache-host from other tools.
+
+Lets a consumer (e.g. bty) point downloads at withcache without re-implementing
+the ``/b/`` URL scheme. Stdlib only, so importing it pulls in no third-party
+dependencies.
+
+    from withcache import client
+
+    # "use the cache when it's warm, the origin otherwise"
+    url = client.serve_url("http://cache:3000", origin) or origin
+
+The ``/b/<urlsafe-b64(origin)>/<basename>`` encoding is shared with the shims
+and the server (one definition in :mod:`withcache._shim`), so consumers stay in
+lockstep with the cache-host automatically.
+"""
+
+from __future__ import annotations
+
+import urllib.error
+import urllib.request
+
+from . import _shim
+
+__all__ = ["PROBE_TIMEOUT", "blob_url", "cache_base", "is_cached", "serve_url"]
+
+PROBE_TIMEOUT = 3.0  # seconds; never block the caller on a slow/unreachable cache
+
+#: Normalize a server value: accepts 'host', 'host:3000', or 'http://host:3000'.
+cache_base = _shim.cache_base
+
+
+def blob_url(server: str, origin: str) -> str:
+    """The cache-host serve URL for ``origin``:
+    ``<server>/b/<urlsafe-b64(origin), unpadded>/<basename>``. The trailing
+    basename is cosmetic (so any downloader names the saved file after the
+    artifact); the cache keys on the decoded origin URL."""
+    return _shim.blob_url(_shim.cache_base(server), origin)
+
+
+def is_cached(
+    server: str,
+    origin: str,
+    timeout: float = PROBE_TIMEOUT,
+    headers: dict[str, str] | None = None,
+) -> bool:
+    """True if the cache-host already holds ``origin`` (a ``HEAD`` on ``/b/``
+    returns 200). A miss (404), an unreachable host, a timeout, or any error
+    returns False, so a caller can safely fall back to the origin. The HEAD
+    also *warms* an auto-fetch cache-host: the miss is recorded and the
+    background fill enqueued, so a later probe flips to cached.
+
+    ``headers`` (optional) attaches request headers to the HEAD. The
+    cache-host forwards a client-supplied ``Authorization`` into its
+    background-fetch worker, so a consumer that has just minted an OCI
+    bearer (the typical use case: bty resolving an ``oras://`` catalog
+    entry to a ``ghcr.io`` blob URL at import time) can warm the cache
+    against that token-gated origin in one probe. Other entries in
+    ``headers`` round-trip the same way; only ``Authorization`` is
+    forwarded into the fetch on the server side.
+    """
+    req = urllib.request.Request(blob_url(server, origin), method="HEAD")
+    if headers:
+        for k, v in headers.items():
+            req.add_header(k, v)
+    try:
+        with urllib.request.urlopen(req, timeout=timeout) as resp:
+            return bool(resp.status == 200)
+    except urllib.error.HTTPError:
+        return False  # 404 miss (now recorded + enqueued by the cache-host)
+    except (urllib.error.URLError, OSError):
+        return False  # unreachable / timeout -> caller serves the origin itself
+
+
+def serve_url(
+    server: str,
+    origin: str,
+    timeout: float = PROBE_TIMEOUT,
+    headers: dict[str, str] | None = None,
+) -> str | None:
+    """The cache-host serve URL for ``origin`` if the cache holds it, else
+    ``None`` -- the convenience form of "use the cache when warm":
+
+        url = client.serve_url(cache, origin) or origin
+
+    ``headers`` is passed through to :func:`is_cached` for the HEAD probe;
+    the returned serve URL never carries auth (cached bytes are served
+    without revisiting the origin).
+    """
+    return blob_url(server, origin) if is_cached(server, origin, timeout, headers=headers) else None

{withcache-0.2.0 → withcache-0.4.0}/src/withcache/server.py

@@ -1,16 +1,17 @@
 #!/usr/bin/env python3
-"""withcache cache-host — an operator-curated artifact cache.
+"""withcache cache-host — a URL-keyed artifact cache.
 
 Stdlib only (http.server + sqlite3 + urllib). Serves cached blobs keyed by
-their origin URL. A cache miss is *not* fetched automatically: it is recorded
-in a miss table so an operator can review it and press "Download", at which
-point the cache-host pulls the artifact from origin and stores it. There is
-also an "add from URI" form to pre-seed an artifact before anyone misses it.
+their origin URL. By default a cache miss is auto-fetched: it is recorded in the
+miss table and pulled from origin in the background, so the next request hits
+(the client falls through to origin on the first miss). Run with `--curate` to
+require an operator to approve each pull via a small web UI instead; either way
+you can pre-seed an artifact with the "Add from URI" form.
 
 This is the only component that needs internet egress (and any vendor creds).
 Clients never write to it.
 
-Auth (modelled on bty's single-tenant approach, minus PAM): the read path
+Auth (single-tenant: env password + signed cookie): the read path
 (`/blob`, `/healthz`) is open so clients never log in; the operator surface
 (`/` and `/admin/*`) is gated behind a server-signed session cookie. Login at
 `POST /ui/login` checks the password in $WITHCACHE_ADMIN_PASSWORD and flips the
@@ -60,6 +61,28 @@ def human_size(n: int) -> str:
     return f"{n} B"
 
 
+def parse_size(s: str) -> int:
+    """Parse '0', '1024', '50M', '20G', '1.5T' into bytes (suffixes are 1024-based)."""
+    s = str(s).strip()
+    if not s:
+        return 0
+    units = {"K": 1024, "M": 1024**2, "G": 1024**3, "T": 1024**4}
+    if s[-1].upper() in units:
+        return int(float(s[:-1]) * units[s[-1].upper()])
+    return int(s)
+
+
+def parse_headers(raw: str) -> dict | None:
+    """Parse 'Name: Value' lines (e.g. a registry Authorization header that bty
+    pre-resolves for an oras blob) into a dict for the origin fetch; None if empty."""
+    out = {}
+    for line in (raw or "").splitlines():
+        name, sep, value = line.partition(":")
+        if sep and name.strip():
+            out[name.strip()] = value.strip()
+    return out or None
+
+
 # --------------------------------------------------------------------------
 # Auth — server-signed session cookie (bty-style, env-password instead of PAM)
 # --------------------------------------------------------------------------
@@ -135,12 +158,13 @@ class Auth:
 class Store:
     """Blobs on disk keyed by hash(normalized url); metadata in SQLite."""
 
-    def __init__(self, data_dir: str, keep_query: bool):
+    def __init__(self, data_dir: str, keep_query: bool, max_bytes: int = 0):
         self.data_dir = os.path.abspath(data_dir)
         self.blob_dir = os.path.join(self.data_dir, "blobs")
         self.tmp_dir = os.path.join(self.data_dir, "tmp")
         self.db_path = os.path.join(self.data_dir, "cache.db")
         self.keep_query = keep_query
+        self.max_bytes = max_bytes  # cap on total cached bytes; 0 = unlimited
         os.makedirs(self.blob_dir, exist_ok=True)
         os.makedirs(self.tmp_dir, exist_ok=True)
         self._init_db()
@@ -217,6 +241,15 @@ class Store:
             m = c.execute("SELECT COUNT(*) FROM misses").fetchone()[0]
         return b, m
 
+    def total_size(self) -> int:
+        with self.conn() as c:
+            return c.execute("SELECT COALESCE(SUM(size), 0) FROM blobs").fetchone()[0]
+
+    def has_capacity(self) -> bool:
+        """False once stored bytes reach --max-bytes (0 = unlimited). The guard
+        refuses *new* fills when full; it never evicts (delete is manual)."""
+        return self.max_bytes <= 0 or self.total_size() < self.max_bytes
+
     # -- writes ------------------------------------------------------------
     def record_miss(self, url: str):
         key = self.key_of(self.normalize(url))
@@ -243,17 +276,34 @@ class Store:
         with _DB_WRITE_LOCK, self.conn() as c:
             c.execute("DELETE FROM misses WHERE key=?", (key,))
 
-    def store_from_origin(self, url: str, progress=None, cancel=None) -> sqlite3.Row:
+    def delete_blob(self, key: str):
+        """Drop a cached artifact (row + bytes). The manual half of eviction."""
+        with _DB_WRITE_LOCK, self.conn() as c:
+            c.execute("DELETE FROM blobs WHERE key=?", (key,))
+        try:
+            os.remove(self.blob_path(key))
+        except FileNotFoundError:
+            pass
+
+    def store_from_origin(self, url: str, progress=None, cancel=None, headers=None) -> sqlite3.Row:
         """Operator-triggered: pull the artifact from origin and store it.
 
         ``progress(done, total)`` is called as bytes arrive (total may be None);
         ``cancel()`` is polled between chunks and, if truthy, aborts the pull
         with :class:`DownloadCancelled` and leaves no partial file behind.
+        ``headers`` adds request headers to the origin fetch (e.g. a registry
+        bearer token bty pre-resolved for an oras blob). Raises :class:`CacheFull`
+        if the cache is already at --max-bytes.
         """
+        if not self.has_capacity():
+            raise CacheFull(f"cache full (>= {self.max_bytes} bytes); refusing to fetch {url}")
         normalized = self.normalize(url)
         key = self.key_of(normalized)
         tmp = os.path.join(self.tmp_dir, key + ".part")
-        req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
+        req_headers = {"User-Agent": USER_AGENT}
+        if headers:
+            req_headers.update(headers)
+        req = urllib.request.Request(url, headers=req_headers)
         sha = hashlib.sha256()
         size = 0
         try:
@@ -315,6 +365,10 @@ class DownloadCancelled(Exception):
     """Raised inside a worker when its job's cancel flag is set."""
 
 
+class CacheFull(Exception):
+    """Raised when --max-bytes is reached; the fill is refused, not evicted."""
+
+
 @dataclass
 class Job:
     id: int
@@ -326,6 +380,7 @@ class Job:
     finished_at: float | None = None
     error: str | None = None
     sha256: str | None = None
+    headers: dict | None = field(default=None, repr=False)  # e.g. registry auth; never logged
     _cancel: threading.Event = field(default_factory=threading.Event, repr=False)
 
 
@@ -345,12 +400,12 @@ class DownloadManager:
         for _ in range(max(1, workers)):
             threading.Thread(target=self._worker, daemon=True).start()
 
-    def enqueue(self, url: str) -> Job:
+    def enqueue(self, url: str, headers: dict | None = None) -> Job:
         with self._lock:
             jid = self._active.get(url)
             if jid is not None and self._jobs[jid].status in PENDING_STATES:
                 return self._jobs[jid]  # dedup an already-pending pull
-            job = Job(id=next(self._ids), url=url)
+            job = Job(id=next(self._ids), url=url, headers=headers)
             self._jobs[job.id] = job
             self._active[url] = job.id
         self._q.put(job.id)
@@ -392,6 +447,7 @@ class DownloadManager:
                     job.url,
                     progress=lambda done, total, j=job: _set_progress(j, done, total),
                     cancel=job._cancel.is_set,
+                    headers=job.headers,
                 )
                 with self._lock:
                     job.status = "completed"
@@ -474,7 +530,13 @@ class Handler(http.server.BaseHTTPRequestHandler):
         else:
             self.send_text(404, "")
 
-    ADMIN_POST = ("/admin/fetch", "/admin/dismiss", "/admin/cancel", "/admin/clear")
+    ADMIN_POST = (
+        "/admin/fetch",
+        "/admin/dismiss",
+        "/admin/delete",
+        "/admin/cancel",
+        "/admin/clear",
+    )
 
     def do_POST(self):
         parsed = urllib.parse.urlsplit(self.path)
@@ -490,9 +552,11 @@ class Handler(http.server.BaseHTTPRequestHandler):
             if parsed.path == "/admin/fetch":
                 url = form.get("url", "").strip()
                 if url:
-                    self.mgr.enqueue(url)
+                    self.mgr.enqueue(url, headers=parse_headers(form.get("header", "")))
             elif parsed.path == "/admin/dismiss":
                 self.store.dismiss(form.get("key", "").strip())
+            elif parsed.path == "/admin/delete":
+                self.store.delete_blob(form.get("key", "").strip())
             elif parsed.path == "/admin/cancel":
                 jid = form.get("id", "")
                 if jid.isdigit():
@@ -594,11 +658,27 @@ class Handler(http.server.BaseHTTPRequestHandler):
         row = self.store.get_blob(url)
         if row is None:
             self.store.record_miss(url)
-            if self.auto_fetch:
+            if self.auto_fetch and self.store.has_capacity():
                 # Pull it in the background so the next request hits; the client
-                # gets this one from origin (the shim falls through on a miss).
-                # In --curate mode an operator triggers the pull instead.
-                self.mgr.enqueue(url)
+                # gets this one from origin (the shim, or bty's fallback chain,
+                # falls through on a miss). In --curate mode an operator triggers
+                # the pull instead; when the cache is full we record the miss but
+                # schedule nothing (delete something first).
+                #
+                # Forward the client's ``Authorization`` header into the worker
+                # job so a token-gated origin (typical use case: a fresh OCI
+                # bearer on a ghcr.io blob URL minted by bty-web at catalog
+                # import time) can be fetched. Without this the worker runs
+                # anonymous and 401s; the URL stays uncached forever. Keep the
+                # allowlist narrow on purpose: ``Authorization`` is the only
+                # request header we proxy onto the worker. The ``/admin/fetch``
+                # operator endpoint still carries its own ``headers=`` payload
+                # for the curated path.
+                fwd_headers = None
+                auth = self.headers.get("Authorization")
+                if auth:
+                    fwd_headers = {"Authorization": auth}
+                self.mgr.enqueue(url, headers=fwd_headers)
             self.send_text(404, "cache miss (recorded)\n")
             return
         path = self.store.blob_path(row["key"])
@@ -736,6 +816,10 @@ class Handler(http.server.BaseHTTPRequestHandler):
         jobs = self.mgr.list()
         misses = self.store.list_misses()
         blobs = self.store.list_blobs()
+        used = human_size(self.store.total_size())
+        if self.store.max_bytes:
+            used += f" / {human_size(self.store.max_bytes)}"
+        full = "" if self.store.has_capacity() else " &middot; <strong>cache full</strong>"
 
         job_rows = (
             "".join(self._job_row(j) for j in jobs)
@@ -774,14 +858,21 @@ class Handler(http.server.BaseHTTPRequestHandler):
                 <td class="num">{b["misses"]}</td>
                 <td class="mono">{html.escape(b["sha256"][:12])}…</td>
                 <td><small>{html.escape(b["fetched_at"])}</small></td>
+                <td>
+                  <form hx-post="/admin/delete" hx-target="#dash" hx-swap="innerHTML"
+                        hx-confirm="Delete this cached artifact?">
+                    <input type="hidden" name="key" value="{html.escape(b["key"], quote=True)}">
+                    <button type="submit" class="secondary outline">Delete</button>
+                  </form>
+                </td>
             </tr>"""
                 for b in blobs
             )
-            or '<tr><td colspan="6"><em>Cache is empty.</em></td></tr>'
+            or '<tr><td colspan="7"><em>Cache is empty.</em></td></tr>'
         )
 
         return f"""
-  <p><small>{nblobs} cached &middot; {nmisses} pending miss(es)</small></p>
+  <p><small>{nblobs} cached ({used}){full} &middot; {nmisses} pending miss(es)</small></p>
 
   <div class="row">
     <h4>Downloads</h4>
@@ -805,7 +896,7 @@ class Handler(http.server.BaseHTTPRequestHandler):
   <figure><table class="striped">
     <thead><tr>
       <th>URL</th><th>Size</th><th class="num">Hits</th><th class="num">Misses</th>
-      <th>SHA-256</th><th>Fetched</th>
+      <th>SHA-256</th><th>Fetched</th><th>Action</th>
     </tr></thead>
     <tbody>{blob_rows}</tbody>
   </table></figure>"""
@@ -869,9 +960,15 @@ def main():
         help="require an operator to approve each pull (default: auto-fetch a "
         "missed artifact in the background so the next request hits)",
     )
+    ap.add_argument(
+        "--max-bytes",
+        default="0",
+        help="cap total cached bytes and refuse new fills when full (0 = "
+        "unlimited; accepts 1024-based suffixes, e.g. 50G). Eviction is manual.",
+    )
     args = ap.parse_args()
 
-    store = Store(args.data_dir, keep_query=args.keep_query)
+    store = Store(args.data_dir, keep_query=args.keep_query, max_bytes=parse_size(args.max_bytes))
     auth = Auth(resolve_secret(store.data_dir), os.environ.get("WITHCACHE_ADMIN_PASSWORD"))
     mgr = DownloadManager(store, workers=args.workers)
 
@@ -883,7 +980,8 @@ def main():
     print(
         f"withcache cache-host on http://{args.host}:{args.port}  "
         f"(data={store.data_dir}, keep_query={args.keep_query}, workers={args.workers}, "
-        f"mode={'curate' if args.curate else 'auto-fetch'})",
+        f"mode={'curate' if args.curate else 'auto-fetch'}, "
+        f"max_bytes={'unlimited' if not store.max_bytes else human_size(store.max_bytes)})",
         flush=True,
     )
     if not auth.enabled:

{withcache-0.2.0 → withcache-0.4.0}/tests/test_withcache.py

@@ -11,6 +11,7 @@ import socketserver
 import sys
 import tempfile
 import threading
+import time
 import unittest
 
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "src"))
@@ -19,7 +20,7 @@ import base64  # noqa: E402
 import urllib.error  # noqa: E402
 import urllib.request  # noqa: E402
 
-from withcache import _shim, curlwithcache, server, wgetwithcache  # noqa: E402
+from withcache import _shim, client, curlwithcache, server, wgetwithcache  # noqa: E402
 
 
 # --------------------------------------------------------------------------
@@ -148,6 +149,23 @@ class TestStoreFromOrigin(unittest.TestCase):
         got = self.store.get_blob(url)
         self.assertEqual((got["hits"], got["misses"]), (2, 2))
 
+    def test_delete_blob_removes_row_and_file(self):
+        url = f"http://127.0.0.1:{self.port}/artifact.bin"
+        row = self.store.store_from_origin(url)
+        path = self.store.blob_path(row["key"])
+        self.assertTrue(os.path.exists(path))
+        self.store.delete_blob(row["key"])
+        self.assertIsNone(self.store.get_blob(url))
+        self.assertFalse(os.path.exists(path))
+
+    def test_capacity_guard_refuses_new_fills_when_full(self):
+        store = server.Store(tempfile.mkdtemp(), keep_query=False, max_bytes=1)
+        self.assertTrue(store.has_capacity())  # empty: room for the first
+        store.store_from_origin(f"http://127.0.0.1:{self.port}/a.bin")
+        self.assertFalse(store.has_capacity())  # now over the 1-byte cap
+        with self.assertRaises(server.CacheFull):
+            store.store_from_origin(f"http://127.0.0.1:{self.port}/b.bin")
+
 
 # --------------------------------------------------------------------------
 # _shim: URL detection, rewrite, real-tool resolution, env, path-encoding
@@ -422,5 +440,215 @@ class TestAutoFetchOnMiss(unittest.TestCase):
             httpd.server_close()
 
 
+# --------------------------------------------------------------------------
+# Fetch-with-headers: a registry blob behind bearer auth (the oras case). bty
+# pre-resolves the token and hands it to withcache for the fill.
+# --------------------------------------------------------------------------
+class _AuthOrigin(http.server.BaseHTTPRequestHandler):
+    TOKEN = "Bearer s3cret"
+
+    def do_GET(self):
+        if self.headers.get("Authorization") != self.TOKEN:
+            self.send_response(401)
+            self.send_header("Content-Length", "0")
+            self.end_headers()
+            return
+        self.send_response(200)
+        self.send_header("Content-Length", str(len(PAYLOAD)))
+        self.end_headers()
+        self.wfile.write(PAYLOAD)
+
+    def log_message(self, format, *args):
+        pass
+
+
+class TestFetchWithHeaders(unittest.TestCase):
+    def setUp(self):
+        self.httpd = socketserver.TCPServer(("127.0.0.1", 0), _AuthOrigin)
+        threading.Thread(target=self.httpd.serve_forever, daemon=True).start()
+        self.url = f"http://127.0.0.1:{self.httpd.server_address[1]}/blob.bin"
+        self.store = server.Store(tempfile.mkdtemp(), keep_query=False)
+
+    def tearDown(self):
+        self.httpd.shutdown()
+        self.httpd.server_close()
+
+    def test_fetch_without_header_is_rejected(self):
+        with self.assertRaises(urllib.error.HTTPError) as cm:
+            self.store.store_from_origin(self.url)
+        self.assertEqual(cm.exception.code, 401)
+
+    def test_fetch_with_bearer_header_succeeds(self):
+        row = self.store.store_from_origin(self.url, headers={"Authorization": _AuthOrigin.TOKEN})
+        self.assertEqual(row["size"], len(PAYLOAD))
+
+
+# --------------------------------------------------------------------------
+# HEAD with an Authorization header should propagate that header into the
+# auto-fetch worker so a 401-gated origin (e.g. a ghcr.io blob URL behind a
+# bty-minted OCI bearer) actually fills. Without this propagation the worker
+# pulls anonymous, the origin 401s, and the URL stays uncached forever.
+# --------------------------------------------------------------------------
+class TestHeadForwardsAuthorizationToAutoFetch(unittest.TestCase):
+    def setUp(self):
+        self.origin = socketserver.TCPServer(("127.0.0.1", 0), _AuthOrigin)
+        threading.Thread(target=self.origin.serve_forever, daemon=True).start()
+        self.origin_url = f"http://127.0.0.1:{self.origin.server_address[1]}/blob.bin"
+        self.httpd, self.store = _start_withcache(auto_fetch=True)
+        self.base = f"http://127.0.0.1:{self.httpd.server_address[1]}"
+
+    def tearDown(self):
+        for s in (self.origin, self.httpd):
+            s.shutdown()
+            s.server_close()
+
+    def _wait_for_fill(self, timeout_s=2.0):
+        deadline = time.monotonic() + timeout_s
+        while time.monotonic() < deadline:
+            if self.store.get_blob(self.origin_url) is not None:
+                return True
+            time.sleep(0.02)
+        return False
+
+    def test_head_with_authorization_triggers_authed_fetch(self):
+        bu = _shim.blob_url(self.base, self.origin_url)
+        req = urllib.request.Request(bu, method="HEAD")
+        req.add_header("Authorization", _AuthOrigin.TOKEN)
+        with self.assertRaises(urllib.error.HTTPError) as cm:
+            urllib.request.urlopen(req)
+        self.assertEqual(cm.exception.code, 404)  # miss; recorded + enqueued
+        # The worker should have fetched in the background using the header.
+        self.assertTrue(
+            self._wait_for_fill(),
+            "expected blob to be cached after auth-bearing HEAD",
+        )
+
+    def test_head_without_authorization_leaves_origin_401_and_cache_empty(self):
+        # Negative: no Authorization on the HEAD means the worker is enqueued
+        # anonymous, the origin 401s, nothing lands. Verifies the new code
+        # path is genuinely opt-in (HEAD without auth keeps the old behaviour).
+        bu = _shim.blob_url(self.base, self.origin_url)
+        with self.assertRaises(urllib.error.HTTPError) as cm:
+            urllib.request.urlopen(urllib.request.Request(bu, method="HEAD"))
+        self.assertEqual(cm.exception.code, 404)
+        self.assertFalse(
+            self._wait_for_fill(timeout_s=0.5),
+            "expected no blob without forwarded auth",
+        )
+
+
+# --------------------------------------------------------------------------
+# Pure helpers
+# --------------------------------------------------------------------------
+class TestParsers(unittest.TestCase):
+    def test_parse_size(self):
+        self.assertEqual(server.parse_size(""), 0)
+        self.assertEqual(server.parse_size("0"), 0)
+        self.assertEqual(server.parse_size("1024"), 1024)
+        self.assertEqual(server.parse_size("50M"), 50 * 1024**2)
+        self.assertEqual(server.parse_size("1.5G"), int(1.5 * 1024**3))
+
+    def test_parse_headers(self):
+        self.assertIsNone(server.parse_headers(""))
+        self.assertEqual(
+            server.parse_headers("Authorization: Bearer x"), {"Authorization": "Bearer x"}
+        )
+        self.assertEqual(server.parse_headers("A: 1\nB: 2"), {"A": "1", "B": "2"})
+
+
+# --------------------------------------------------------------------------
+# Client library: what a consumer (e.g. bty) imports instead of reimplementing
+# the /b/ protocol.
+# --------------------------------------------------------------------------
+class TestClientLibrary(unittest.TestCase):
+    def setUp(self):
+        self.origin = socketserver.TCPServer(("127.0.0.1", 0), _Origin)
+        threading.Thread(target=self.origin.serve_forever, daemon=True).start()
+        self.origin_url = f"http://127.0.0.1:{self.origin.server_address[1]}/art.bin"
+        self.httpd, self.store = _start_withcache()
+        self.base = f"http://127.0.0.1:{self.httpd.server_address[1]}"
+
+    def tearDown(self):
+        for s in (self.origin, self.httpd):
+            s.shutdown()
+            s.server_close()
+
+    def test_blob_url_matches_shim_and_normalizes_server(self):
+        # accepts a host/host:port/http URL and emits the same /b/ URL as the shim
+        self.assertEqual(
+            client.blob_url(self.base, self.origin_url),
+            _shim.blob_url(_shim.cache_base(self.base), self.origin_url),
+        )
+
+    def test_is_cached_and_serve_url_track_the_cache(self):
+        self.assertFalse(client.is_cached(self.base, self.origin_url))
+        self.assertIsNone(client.serve_url(self.base, self.origin_url))
+        self.store.store_from_origin(self.origin_url)  # warm it
+        self.assertTrue(client.is_cached(self.base, self.origin_url))
+        self.assertEqual(
+            client.serve_url(self.base, self.origin_url),
+            client.blob_url(self.base, self.origin_url),
+        )
+
+    def test_is_cached_unreachable_is_false(self):
+        self.assertFalse(client.is_cached("http://127.0.0.1:9", self.origin_url, timeout=0.5))
+
+
+# --------------------------------------------------------------------------
+# Client + server end-to-end: a HEAD with ``headers={"Authorization": ...}``
+# warms the cache against a 401-gated origin. Mirrors the bty oras case
+# (resolved ghcr.io blob URL + freshly-minted OCI bearer).
+# --------------------------------------------------------------------------
+class TestClientLibraryAuthForwarding(unittest.TestCase):
+    def setUp(self):
+        self.origin = socketserver.TCPServer(("127.0.0.1", 0), _AuthOrigin)
+        threading.Thread(target=self.origin.serve_forever, daemon=True).start()
+        self.origin_url = f"http://127.0.0.1:{self.origin.server_address[1]}/blob.bin"
+        self.httpd, self.store = _start_withcache(auto_fetch=True)
+        self.base = f"http://127.0.0.1:{self.httpd.server_address[1]}"
+
+    def tearDown(self):
+        for s in (self.origin, self.httpd):
+            s.shutdown()
+            s.server_close()
+
+    def test_is_cached_with_authorization_warms_auth_gated_origin(self):
+        # Cold: no auth -> background fetch goes anonymous, 401s, cache empty.
+        self.assertFalse(client.is_cached(self.base, self.origin_url))
+        deadline = time.monotonic() + 0.5
+        while time.monotonic() < deadline:
+            if self.store.get_blob(self.origin_url) is not None:
+                break
+            time.sleep(0.02)
+        self.assertIsNone(self.store.get_blob(self.origin_url))
+
+        # Warm-with-token: HEAD carries Authorization; server forwards it
+        # into the fetch worker; the auth-gated origin returns the bytes.
+        self.assertFalse(
+            client.is_cached(
+                self.base,
+                self.origin_url,
+                headers={"Authorization": _AuthOrigin.TOKEN},
+            )
+        )
+        deadline = time.monotonic() + 2.0
+        while time.monotonic() < deadline:
+            if self.store.get_blob(self.origin_url) is not None:
+                break
+            time.sleep(0.02)
+        self.assertIsNotNone(
+            self.store.get_blob(self.origin_url),
+            "expected auth-bearing HEAD to fill the cache via forwarded Authorization",
+        )
+
+        # And once cached, the auth header is no longer needed: a plain HEAD
+        # hits 200, serve_url returns the blob URL without auth.
+        self.assertTrue(client.is_cached(self.base, self.origin_url))
+        self.assertEqual(
+            client.serve_url(self.base, self.origin_url),
+            client.blob_url(self.base, self.origin_url),
+        )
+
+
 if __name__ == "__main__":
     unittest.main(verbosity=2)

withcache-0.2.0/src/withcache/__init__.py

@@ -1,11 +0,0 @@
-"""withcache — operator-curated, URL-keyed artifact cache for a small lab.
-
-Two console entry points (see pyproject.toml):
-  withcache         -> withcache.client:main   (the cache-aware downloader)
-  withcache-server  -> withcache.server:main   (the cache-host)
-
-Both modules are stdlib-only and self-contained, so either file can also be
-copied and run on its own with a plain ``python3``.
-"""
-
-__version__ = "0.2.0"