withcache 0.2.0__py3-none-any.whl

withcache-0.2.0.dist-info/METADATA

@@ -0,0 +1,271 @@
+Metadata-Version: 2.4
+Name: withcache
+Version: 0.2.0
+Summary: Operator-curated, URL-keyed artifact cache for a small lab (CUDA/ROCm/DOCA/firmware)
+Project-URL: Homepage, https://github.com/safl/withcache
+Author-email: "Simon A. F. Lund" <safl@safl.dk>
+License: BSD-3-Clause
+License-File: LICENSE
+Keywords: artifacts,cache,cuda,doca,firmware,lab,rocm
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: System Administrators
+Classifier: License :: OSI Approved :: BSD License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: System :: Archiving :: Mirroring
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+
+# withcache
+
+[![ci](https://github.com/safl/withcache/actions/workflows/ci.yml/badge.svg)](https://github.com/safl/withcache/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/withcache.svg)](https://pypi.org/project/withcache/)
+[![license](https://img.shields.io/badge/license-BSD--3--Clause-blue.svg)](LICENSE)
+[![built with Zig](https://img.shields.io/badge/built%20with-Zig%200.16.0-f7a41d.svg)](https://ziglang.org)
+[![static musl](https://img.shields.io/badge/static%20musl-x86__64%20%7C%20aarch64-blue.svg)](https://github.com/safl/withcache/releases)
+
+A tiny, **operator-curated** artifact cache for a small lab, for the big vendor
+downloads you re-pull constantly (CUDA, ROCm, DOCA, firmware, drivers), fronted
+by **transparent `curl`/`wget` shims** so existing scripts use it with no changes.
+
+Think of it as **"`ccache` for HTTP artifacts, without a proxy."**
+
+```
+curl -fsSL https://the/origin/cuda.tar.gz -o cuda.tar.gz     # your script, unchanged
+   └─ curlwithcache shim ─ WITHCACHE_SERVER set?
+        ├─ cached  → served from the cache-host (fast, local)
+        └─ miss/unset/unreachable → runs the real curl, exactly as written
+```
+
+Artifacts are cached **by their origin URL as a key**; the shim opts in by
+re-pointing the URL at the cache. No transparent proxy, no TLS interception, no
+client CA. The URL is a lookup key, not a connection target.
+
+By default a miss is **auto-fetched**: the request falls through to origin (so
+the caller gets its file straight away), and the cache-host pulls the same
+artifact in the background, so the next request hits. Run with **`--curate`** to
+require a human instead, who reviews the miss list in a small web UI and presses
+**Download** (or pre-seeds via *Add from URI*). Either way the cache-host is the
+only box that needs internet egress (and any vendor credentials), and clients
+never write to it.
+
+## Why not just curl + a caching proxy?
+
+For `https://` (i.e. every vendor download) a forward proxy can't cache without
+**SSL-bump / MITM**: curl tunnels TLS end-to-end via `CONNECT`, so the proxy
+only sees ciphertext. The shim sidesteps that entirely by *re-pointing the URL*
+to the cache instead of intercepting the connection. And no proxy offers the
+optional **operator-curated** model (`--curate`: a miss queue a human approves).
+
+## Components
+
+| Path                          | What it is                                                  |
+|-------------------------------|-------------------------------------------------------------|
+| `src/withcache/server.py`     | The cache-host: blob store + miss table + **background download manager** + operator UI (Pico.css + HTMX) |
+| `src/withcache/_shim.py`      | Shared shim core (find URL → probe → rewrite → exec)        |
+| `src/withcache/curlwithcache.py` / `wgetwithcache.py` | The Python `curl` / `wget` shims    |
+| `shim/shim.zig`               | The native shim: one static binary, both tools via `argv[0]` |
+| `deploy/Containerfile`, `deploy/compose.yml` | Single Podman/Docker host deploy             |
+
+The cache-host and the Python shims are **stdlib-only** (no third-party runtime
+deps); the native shim is a dependency-free static binary.
+
+## Install
+
+The **cache-host** and **Python shims** (works on any box with Python):
+
+```sh
+pipx install withcache    # or: uv tool install withcache  /  pip install withcache
+# provides: curlwithcache  wgetwithcache  withcache-server
+```
+
+The **native shim** (no Python needed, for minimal/distroless boxes; ~200 KB
+static musl binary). Grab it from the [Releases] page; one binary serves both
+tools by the name it's invoked as:
+
+```sh
+curl -L .../releases/.../withcache-shim-x86_64-linux-musl -o /usr/local/bin/curlwithcache
+chmod +x /usr/local/bin/curlwithcache
+```
+
+The Python shim is also the tested **oracle** and install-time fallback for
+platforms without a prebuilt binary; a [differential test](tests/test_differential.py)
+asserts the binary and the Python `plan()` rewrite argv identically.
+
+[Releases]: https://github.com/safl/withcache/releases
+[direnv]: https://direnv.net
+
+## Deploy the cache-host
+
+```sh
+export WITHCACHE_ADMIN_PASSWORD=change-me    # protects the operator UI
+podman compose -f deploy/compose.yml up -d   # or: docker compose -f ...
+# operator UI:  http://withcache-server:3000/
+```
+
+Or without containers:
+
+```sh
+WITHCACHE_ADMIN_PASSWORD=change-me withcache-server --data-dir ./data --port 3000
+```
+
+Data (blobs + `cache.db` + `session-secret`) lives in the `/data` volume (or
+`--data-dir`). Artifacts are immutable per version, so there's no cache
+invalidation. `--workers N` sets the number of concurrent download workers, and
+`--curate` switches from auto-fetch to operator-approved pulls.
+
+## Use the shims (transparent `curl` / `wget`)
+
+Every approach is the same two ingredients: (1) point at the cache with
+`WITHCACHE_SERVER`, and (2) make `curl`/`wget` resolve to the shim. They differ
+only in **how widely the system `curl`/`wget` is shadowed**. Pick the least
+invasive one that fits.
+
+> **Safety:** with `WITHCACHE_SERVER` unset the shim is a pure pass-through (it
+> just `exec`s the real tool, zero network/parsing), so even the system-wide
+> setup is harmless wherever the cache isn't configured. Worst case is always
+> "no caching, `curl` still works."
+
+These all use `command -v curlwithcache`, so they work whether you installed the
+native binary or the Python launcher (both land under that name).
+
+### 1. No shadowing: call the shims by name (least invasive)
+
+Nothing is renamed; you opt in per command. Good for trying it out or a script
+you can edit.
+
+```sh
+export WITHCACHE_SERVER=http://withcache-server:3000
+curlwithcache -fsSL https://the/origin/cuda.tar.gz -o cuda.tar.gz
+wgetwithcache https://the/origin/rocm.tar.gz
+```
+
+### 2. This shell only: shadow `curl`/`wget` for the session
+
+Put `curl`/`wget` symlinks in a dir and prepend it to `PATH` in the current
+shell. Reversible by just closing the shell.
+
+```sh
+mkdir -p ~/.withcache/bin
+ln -sf "$(command -v curlwithcache)" ~/.withcache/bin/curl
+ln -sf "$(command -v wgetwithcache)" ~/.withcache/bin/wget
+
+export WITHCACHE_SERVER=http://withcache-server:3000
+export PATH="$HOME/.withcache/bin:$PATH"
+hash -r                       # forget any cached curl/wget location
+
+command -v curl               # -> ~/.withcache/bin/curl  (verify it's the shim)
+curl -fsSL https://the/origin/cuda.tar.gz -o cuda.tar.gz   # existing scripts, unchanged
+wget https://the/origin/rocm.tar.gz                        # still saved as rocm.tar.gz
+```
+
+### 3. Your user: make it the default for your shells (persistent)
+
+Create the symlinks once, then add the two exports to your shell rc. Affects all
+your future interactive shells; undo by deleting the block.
+
+```sh
+mkdir -p ~/.withcache/bin
+ln -sf "$(command -v curlwithcache)" ~/.withcache/bin/curl
+ln -sf "$(command -v wgetwithcache)" ~/.withcache/bin/wget
+
+cat >> ~/.bashrc <<'EOF'
+
+# withcache: transparent curl/wget caching
+export WITHCACHE_SERVER=http://withcache-server:3000
+export PATH="$HOME/.withcache/bin:$PATH"
+EOF
+```
+
+### 4. One project only: scope it with direnv
+
+Drop an `.envrc` in a project tree (requires [direnv]); caching applies only
+inside that directory.
+
+```sh
+# .envrc
+export WITHCACHE_SERVER=http://withcache-server:3000
+PATH_add ~/.withcache/bin        # assumes the symlinks from approach 2/3 exist
+```
+
+Then `direnv allow`.
+
+### 5. The whole machine: every user, every shell (most invasive)
+
+Install the shim as `curl`/`wget` in `/usr/local/bin` (ahead of `/usr/bin` on
+the default `PATH`) and set the server globally. This also catches build tools
+and package managers that shell out to `curl`/`wget`.
+
+```sh
+sudo ln -sf "$(command -v curlwithcache)" /usr/local/bin/curl
+sudo ln -sf "$(command -v wgetwithcache)" /usr/local/bin/wget
+
+# A login-shell env file (covers interactive logins; daemons started outside a
+# login shell won't see it; set WITHCACHE_SERVER in their unit if you need it).
+echo 'export WITHCACHE_SERVER=http://withcache-server:3000' \
+  | sudo tee /etc/profile.d/withcache.sh >/dev/null
+```
+
+On minimal/distroless hosts use the [native shim binary](#install) here: same
+symlink, no Python required.
+
+### Verify / turn it off
+
+```sh
+command -v curl                       # which curl is in effect (the shim, or the real one)
+export REAL_CURL=/usr/bin/curl        # optional: pin the wrapped tool (also $REAL_WGET)
+
+unset WITHCACHE_SERVER                 # instantly back to plain curl (pass-through)
+rm ~/.withcache/bin/curl ~/.withcache/bin/wget   # remove shadowing entirely
+```
+
+How it works: the shim **scans for the URL, asks the cache, and execs the real tool**:
+
+1. Find the real `curl`/`wget` on `$PATH` (skipping itself; `$REAL_CURL`/`$REAL_WGET` override).
+2. With `WITHCACHE_SERVER` set, find the URL (the `scheme://` arg, or `--url`).
+3. Probe the cache with that same tool (`curl -I` / `wget --spider`).
+   - **Hit** → re-point only the URL at `http://server/b/<base64(origin)>/<basename>` and `exec` the real tool (so `-o`, `-O`, `-L`, `--retry`, … all still apply, and the file is named after the artifact).
+   - **Miss / unreachable** → `exec` the real tool with your **arguments untouched** (origin); the miss is recorded for the operator.
+4. With no `WITHCACHE_SERVER`, it does **zero** network/parsing, just `exec`s the real tool.
+
+Notes & limits (all degrade gracefully; worst case is "no caching, curl still works"):
+- Needs the wrapped tool present (it shims it). Adds ~Python-startup latency per call.
+- URLs hidden in a `-K`/`-i` config file or piped via stdin aren't seen → those calls pass through uncached.
+- Per-tool env override: `CURLWITHCACHE_SERVER` / `WGETWITHCACHE_SERVER` beat `WITHCACHE_SERVER`.
+
+## Operator UI
+
+`http://withcache-server:3000/` (Pico.css + HTMX, bundled offline) shows:
+- **Misses**: auto-fetched by default, or (under `--curate`) each with **Download** (queues a background pull) and **Dismiss**.
+- **Downloads**: live progress bars, `queued/running/completed/cancelled/failed`, **Cancel**, and **Clear finished**. Downloads run in a background worker pool, not in the request, so large pulls never block, modelled on [bty]'s job managers.
+- **Cached artifacts**: URL, size, **hits** (times served) and **misses** (times requested before it was cached), SHA-256, fetched-at.
+- **Add from URI**: pre-seed an artifact before anyone misses it.
+
+## Auth
+
+Single-tenant session-cookie auth (modelled on [bty]'s approach, env password
+instead of PAM). The **read path** (`/blob`, `/b/…`, `/healthz`) is open so shims
+never log in; the **operator surface** (`/`, `/admin/*`) is gated.
+
+| Env var                    | Purpose                                                  |
+|----------------------------|----------------------------------------------------------|
+| `WITHCACHE_SERVER`         | Cache-host URL the shims use                             |
+| `CURLWITHCACHE_SERVER` / `WGETWITHCACHE_SERVER` | Per-tool override of the above       |
+| `WITHCACHE_ADMIN_PASSWORD` | Operator login password (unset ⇒ UI open, with a warning) |
+| `WITHCACHE_SESSION_SECRET` | Override the persisted cookie-signing key (optional)     |
+
+[bty]: https://github.com/safl/bty
+
+## Cache keys & signed URLs
+
+The key is `scheme://host/path` with the **query string dropped** by default, so
+CDN/presigned URLs (whose tokens change every request) still match by path. Pass
+`--keep-query` to the server for query-sensitive keys. Package-manager repos
+(`.deb`/`.rpm`) are GPG-signed and verified by the client regardless of
+transport, so caching them this way is safe.
+
+## Tests
+
+```sh
+python -m unittest discover -s tests   # stdlib only, no test deps
+```

withcache-0.2.0.dist-info/RECORD

@@ -0,0 +1,14 @@
+withcache/__init__.py,sha256=UrNawCAnRv-ZQxFhqGekxYryydYonSi6ZonQyYhFN-M,421
+withcache/_shim.py,sha256=_CnLxWNmqfXq9QZTqQ6_NFkWxWu97Sp4nJPAQxj39xc,5165
+withcache/curlwithcache.py,sha256=-lFuiVtM8UKQdMP4NPX2TtJAlOfeObHgYvpQpDBpc7k,1796
+withcache/server.py,sha256=DpioGl34bW1Zg0-5WWECSKK_pErMxIqGJ4Q-N3mSFIk,35438
+withcache/wgetwithcache.py,sha256=ZrRKXh91KAajFSsGZnNuNchRmspevTY2bVgkQdwZaaY,1741
+withcache/static/htmx.min.js,sha256=4gndpcgjVHnzFm3vx3UOHbzVpcGAi3eS_C5nM3aPtEc,50917
+withcache/static/pico.min.css,sha256=3V_VWRr9ge4h3MEXrYXAFNw_HxncLXt9EB6grMKSdMI,82194
+withcache-0.2.0.data/scripts/curlwithcache,sha256=g_bsxFt5xobsFCwsuwj62zsQQJa94BrLG7ttm_qlI4A,78
+withcache-0.2.0.data/scripts/wgetwithcache,sha256=QK8imdXLJgxQSyyXIrFJEsdm_IOxH2mq5WAnApPD4xg,78
+withcache-0.2.0.dist-info/METADATA,sha256=nefLMRE29sOa8qpi2RJVDkpxHc55K8W_89euTt0a8mg,12405
+withcache-0.2.0.dist-info/WHEEL,sha256=mffPy8wBnZQn2VnJUU5jE99KsxaSfiyMHV9Yt0aLVxs,87
+withcache-0.2.0.dist-info/entry_points.txt,sha256=ZpOEQxrJ50tSzLARCoEu80pYsWkI11f10EUytK82XIk,59
+withcache-0.2.0.dist-info/licenses/LICENSE,sha256=k28p1vD0tdx5EqOG2L93SQ6EGqXDQTWn4RMptaUyFm0,1503
+withcache-0.2.0.dist-info/RECORD,,

withcache-0.2.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.30.1
+Root-Is-Purelib: true
+Tag: py3-none-any

withcache-0.2.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+withcache-server = withcache.server:main

withcache-0.2.0.dist-info/licenses/LICENSE

@@ -0,0 +1,28 @@
+BSD 3-Clause License
+
+Copyright (c) 2026, Simon A. F. Lund
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.