mlclaw 0.1.0 → 0.2.1

package/assets/hf-tooling/skills/huggingface-spaces/references/requirements.md

@@ -0,0 +1,169 @@
+# requirements.txt for Spaces
+
+Rules for what to pin, what to leave alone, where to source CUDA wheels, and which torch-side-cars drift silently.
+
+## What's preinstalled (do not list)
+
+The Gradio SDK base image already installs these on every hardware tier — listing them in `requirements.txt` causes resolution failures or, worse, lets pip silently drift the runtime out of compatibility:
+
+| Package | Pinning rules |
+|---|---|
+| `gradio` | Don't list. Locked by `sdk_version:` in README frontmatter; pinning here is ignored or breaks. |
+| `spaces` | Don't list. Platform-pinned; a user pin always loses. |
+| `huggingface_hub` | Don't list by default. Pin only as a workaround for old `gradio<5` that imports the removed `HfFolder` symbol (see [`known-errors.md`](known-errors.md)). |
+| `torch` | **Pinnable, but only within `{2.8.0, 2.9.1, 2.10.0, 2.11.0}`.** Anything outside causes `CONFIG_ERROR: torch version in requirements.txt is not compatible`. Default is to leave unpinned (runtime preinstalls 2.11), but pinning is appropriate when (a) a specific version is known-good for your model, (b) you're matching a CUDA-extension wheel's `torch2.X` tag, or (c) a dep would otherwise drag torch outside the supported set. When you pin torch, also pin `torchvision` / `torchaudio` to the matching minor — see the "Torch-family side-car drift" section below. |
+
+## What to list
+
+Everything you actually `import`, including the often-forgotten:
+
+- `torchvision`, `torchaudio` — **not** preinstalled. Leave unpinned; pip resolves against the installed `torch` major.minor.
+- `accelerate` — needed whenever you use `device_map=`. Listing it also silences `low_cpu_mem_usage=False` warnings.
+- `sentencepiece` — required by most LLM tokenizers; rarely transitive.
+- `einops` — required by `flash_attn.layers.rotary` and many model repos.
+- Domain libs: `diffusers`, `transformers`, `safetensors`, `pillow`, `numpy`, etc.
+
+If a research repo ships a Python package directory (`models/`, `pipeline/`, …), just upload the directory with the rest of the Space — the whole repo root is importable as `/home/user/app`. **Do not** try to reference local paths from `requirements.txt`.
+
+## Pinning torch
+
+ZeroGPU accepts only `2.8.0`, `2.9.1`, `2.10.0`, `2.11.0`. Default is unpinned (runtime preinstalls the latest). Pinning is fine — and sometimes warranted — within that set:
+
+- A specific torch is known-good for your model (numerics, attention kernel availability, etc.).
+- A direct-URL CUDA wheel encodes a `torch2.X` tag (see "Prebuilt CUDA wheels" below) — pin torch to match.
+- A dep's `setup.py` would otherwise downgrade torch outside the supported set.
+
+`2.8.0` is the safest fallback for old requirements that refuse modern torch. `2.10.0` / `2.11.0` is the sweet spot for new code. When you pin torch, also pin `torchvision` / `torchaudio` to the matching minor — see the side-car drift section.
+
+When a dep would silently downgrade torch (e.g. some forks of `demucs`, `audiocraft` pin `torchaudio<2.1`), install the offender from `app.py` with `--no-deps` rather than pinning torch around it:
+
+```python
+import subprocess, sys
+subprocess.run([sys.executable, "-m", "pip", "install", "--no-deps",
+                "git+https://github.com/facebookresearch/demucs"], check=True)
+import spaces  # safe now — torch wasn't touched
+```
+
+List the offender's real runtime deps yourself in `requirements.txt`.
+
+## Torch-family side-car drift
+
+`torchvision`, `torchaudio`, `torchcodec` are built against a specific `torch` major.minor. Listing them unpinned **usually** works, but two known drift patterns:
+
+- `torchaudio==2.11.0` (and later) **dropped its `Requires-Dist: torch==X.Y.Z` line**. With torch pinned to 2.10, pip silently resolves torchaudio to 2.11.0 and the import fails on ABI mismatch.
+- `torchcodec` declares no torch dependency in PyPI metadata at all.
+
+Verification after `pip install` or `uv lock --upgrade`:
+
+```bash
+curl -s https://pypi.org/pypi/<pkg>/<version>/json \
+  | python3 -c "import json,sys,re; rd=json.load(sys.stdin)['info'].get('requires_dist') or []; \
+                print('\n'.join(x for x in rd if re.match(r'^torch(?![a-z])', x)) or '(no torch constraint)')"
+```
+
+When PyPI is silent, fall back to the project's README compatibility table (torchcodec's lives at https://github.com/pytorch/torchcodec).
+
+## Prebuilt CUDA wheels — the Blackwell wheels dataset
+
+This is the **first** thing to reach for when a CUDA-extension package has no upstream wheel matching the ZeroGPU torch / CUDA / cxx11-abi cell. Prefer it over any runtime workaround (`pip install git+…` inside `@spaces.GPU`, committed stub packages, `sys.modules` injection, monkey-patch shims), all of which are slower, fragile, and eat `duration` budget. Canonical prebuilt sm_120 wheels:
+
+> https://huggingface.co/datasets/multimodalart/zerogpu-blackwell-wheels
+
+Wheels live at `wheels/<cell>/<wheel>`. A **cell** encodes torch × CUDA × Python as `pt<torch>-cu<cuda>-cp<pyver>`. Every cell ships the same seven packages:
+
+`flash_attn` (two versions: `2.8.3` and `2.7.4.post1`), `xformers`, `pytorch3d`, `nvdiffrast`, `diff_gaussian_rasterization`, `torchmcubes`.
+
+Current cells (12):
+
+| torch | CUDA | Python cells available |
+|---|---|---|
+| 2.8.0 (`pt28`) | 12.8 | cp310, cp311, cp312 |
+| 2.9.1 (`pt291`) | 12.8 | cp310 |
+| 2.10.0 (`pt210`) | 12.8 / 13.0 | cp310 (cu128), cp312 (cu130) |
+| 2.11.0 (`pt211`) | 13.0 | cp312, cp313 |
+| 2.12.0 (`pt212`) | 13.0 | cp310, cp311, cp312, cp313 |
+
+### Picking a cell
+
+1. **Match `cp<pyver>` to your `python_version:`.** The wheels are cp-ABI-specific — a `cp310` wheel needs Python 3.10, `cp312` needs 3.12, etc. (`flash_attn` now ships cp310 **through** cp313, so this is a free choice, not a forced pin to 3.10 as in older versions of this doc.)
+2. **The wheels are torch-minor-tolerant.** The `flash_attn` / `xformers` filenames encode no torch version, so a `pt212-cu130-cp310` wheel runs fine on the live torch-2.11 cp310 runtime. `flash_attn` ships cp310 through cp313, so the Python choice is free. Pick the highest-torch cell for your Python version unless you've pinned an older torch — then match it (torch 2.8 → a `pt28-cu128-cp3XX` cell).
+3. **Copy the *exact* filename from the cell you pick.** The `xformers` build hash differs across cells (`0.0.34+3da0fc92…` on the cu130 / torch≥2.10 cells, `0.0.34+41531cee…` on the cu128 / torch 2.8–2.9 cells). Don't hardcode one filename across cells — list the cell (`hf download --repo-type dataset multimodalart/zerogpu-blackwell-wheels --include "wheels/<cell>/*"` or the Hub file browser) and copy what's there.
+
+Reference by direct URL in `requirements.txt`:
+
+```
+https://huggingface.co/datasets/multimodalart/zerogpu-blackwell-wheels/resolve/main/wheels/<cell>/<wheel>
+```
+
+### Per-package status
+
+Each package's version is constant across cells; only the `cp`/torch/CUDA tags in the filename change. The "instead of" column is the runtime workaround to avoid — the wheel is the clean path.
+
+| Package | Instead of | Version + caveats |
+|---|---|---|
+| `flash_attn` | committing a `flash_attn/` stub package; `sys.modules["flash_attn"] = …` injection | **2.8.3** (default) or **2.7.4.post1** (repos that pin `flash-attn<2.8`). Ships **cp310–cp313**. Needs `einops` for `flash_attn.layers.rotary`. Built `FLASH_ATTN_CUDA_ARCHS=120` (sm_120 only). This is FlashAttention-**2** — FA3/FA4 do **not** run on sm_120 (no TMEM); see [`zerogpu.md`](zerogpu.md) → Attention backends. Its real `flash_attn_2_cuda` also satisfies xformers' `flash_attn_gpu` probe. |
+| `xformers` | an MEA→SDPA monkey-patch shim; a Cutlass-force shim | **0.0.34** (`Requires: torch>=2.10`). **Build hash differs per cell** — copy the exact filename. Auto-dispatch picks FA2 (`fa2F`) on sm_120; classic Cutlass / FA3 reject sm_120 but auto-dispatch never selects them. |
+| `pytorch3d` | a runtime `pip install git+…pytorch3d.git` inside `@spaces.GPU` | **0.7.9**. Needs `numpy`, `iopath`, `fvcore` listed. No torch pin in metadata; loads cleanly on torch 2.11. |
+| `nvdiffrast` | a runtime build with `TORCH_CUDA_ARCH_LIST=12.0` | **0.4.0**. Needs `numpy`. `RasterizeGLContext` is a deprecation alias for `RasterizeCudaContext` — no headless-GL footgun. |
+| `diff_gaussian_rasterization` | a runtime build from `graphdeco-inria/diff-gaussian-rasterization.git` | **Upstream Inria API only** (returns 2-tuple `(color, radii)`). Does NOT match the ashawkey fork (4-tuple incl. alpha+depth) used by `ashawkey/LGM`, `dylanebert/LGM-mini`, etc. Forks need their own wheel. |
+| `torchmcubes` | a runtime `pip install git+…torchmcubes.git` | **0.1.0**. **sm_120 only** (no fatbin for older archs). Works on ZeroGPU / Blackwell; not portable to a dedicated T4 / L4 / A10G Space. |
+
+### Pattern
+
+Resolve the exact filenames from your chosen cell, then:
+
+```
+# requirements.txt  (cell = pt212-cu130-cp310 → needs python_version "3.10")
+numpy
+einops
+https://huggingface.co/datasets/multimodalart/zerogpu-blackwell-wheels/resolve/main/wheels/pt212-cu130-cp310/flash_attn-2.8.3-cp310-cp310-linux_x86_64.whl
+https://huggingface.co/datasets/multimodalart/zerogpu-blackwell-wheels/resolve/main/wheels/pt212-cu130-cp310/xformers-0.0.34+3da0fc92.d20260528-cp39-abi3-linux_x86_64.whl
+```
+
+```yaml
+# README frontmatter — pin Python to match the wheel cell's cp tag
+python_version: "3.10"
+```
+
+**Do not** install these from `@spaces.GPU` startup. A `subprocess.check_call` pip-install at first GPU acquire is strictly worse than the wheel URL — slower cold start, eats `duration` budget, breaks reproducibility, and the build sometimes exceeds the `@spaces.GPU(duration=1500)` cap.
+
+### When you need a wheel that's not in the dataset
+
+Three options, in preference order:
+
+1. **kernels-community** — https://huggingface.co/kernels-community handles ABI matching for you. Often the simplest path; no version pinning needed.
+2. **Upstream wheel matrix** — e.g. flash-attention's releases page ships a fairly complete `cu12 / torch / Python` matrix at https://github.com/Dao-AILab/flash-attention/releases. Pin `torch==X.Y.Z` in `requirements.txt` to match the wheel's `torch2.X` tag.
+3. **Build it yourself and host on HF Hub.** Last resort — see [`debugging.md`](debugging.md) for the in-`@spaces.GPU` source-build pattern as a stopgap while a wheel is being built.
+
+## Reading a CUDA wheel filename
+
+```
+flash_attn-2.8.3+cu130torch2.12cxx11abiFALSE-cp310-cp310-linux_x86_64.whl
+```
+
+| Tag | Meaning |
+|---|---|
+| `cu130` | CUDA major version (13.0) |
+| `torch2.12` | torch major.minor the wheel was compiled against |
+| `cxx11abiFALSE` | C++ stdlib ABI choice (`TRUE` or `FALSE`) |
+| `cp310-cp310` | CPython version (3.10) |
+
+ABI / symbol mismatches at any of these → `ImportError` on first import. Pin `torch` to match `torch2.X`. Set `python_version:` to match `cp3XX`.
+
+## Don't pin `xformers`
+
+Leave bare in `requirements.txt` (or use the prebuilt URL above). Pip picks the wheel matching your installed torch.
+
+## Don't pin `spaces`
+
+Even if a `uv export` produces it, exclude with `--no-emit-package spaces`. The platform always pins its own version.
+
+## Specifically about Python version
+
+Pinning `python_version:` is effectively required:
+
+- ZeroGPU officially supports **3.10.13** and **3.12.12**.
+- The runtime default is 3.10.
+- Pinning to a `cp3XX` wheel matrix (e.g. `cp310` flash_attn wheel) forces matching Python.
+
+Both `"3.12"` and `"3.12.12"` forms are accepted in YAML.

package/assets/hf-tooling/skills/huggingface-spaces/references/zerogpu.md

@@ -0,0 +1,349 @@
+# ZeroGPU
+
+Read this whenever the Space targets ZeroGPU (`zero-a10g` flavor). The SKILL.md's 3-rule summary is a starting point; this file covers the model in enough detail to debug and design.
+
+For numerical limits (per-tier daily quota minutes, runs-per-day caps, current backing GPU, supported Python / torch versions): https://huggingface.co/docs/hub/spaces-zerogpu. Those values change over time and are deliberately kept out of this skill.
+
+## The mental model
+
+A ZeroGPU Space runs as **two processes**:
+
+- **Main web process** — long-lived. Imports `app.py`, launches Gradio. Holds no VRAM and, after the startup "pack" step, no model weights in RAM either.
+- **GPU worker** — short-lived. Forked per `@spaces.GPU` request (or reused if warm). Eventually killed by the ZeroGPU scheduler when another Space needs the slot. Your code never kills its own worker.
+
+`import spaces` monkey-patches `torch.cuda.*` in the main process so that `.to("cuda")` and `torch.cuda.is_available()` work at module scope **without** a real GPU attached. Module-level `model.to("cuda")` is intercepted: the tensor data physically stays in main-process RAM at this point, with a CUDA-presenting "fake" tensor registered alongside. At a startup "pack" step, the backend writes those original CPU tensors to disk via `O_DIRECT` and frees the RAM. After pack, main holds no weights anywhere.
+
+When a `@spaces.GPU` call lands, the scheduler routes it to a worker:
+
+- **Cold worker** — forked from the main process; torch is unpatched; real CUDA is initialized; weights are streamed disk → pinned host → VRAM via a double-buffered pipeline. This is the cold-start cost.
+- **Warm worker** — alive worker bound to the same slot; init is skipped; weights stay on VRAM from the previous call.
+
+A warm worker eventually dies when another Space needs the slot. Occasional cold starts on a low-traffic Space are normal.
+
+## The three rules
+
+### 1. `import spaces` before any CUDA-touching import
+
+```python
+import spaces      # FIRST
+import torch       # then this
+```
+
+If something initializes CUDA before `import spaces`, the patch can't apply and you get `RuntimeError: CUDA has been initialized before importing the spaces package`. For libraries that eagerly init CUDA on import (e.g. `numba.cuda`, NeMo via numba), set the disable env *before* the import:
+
+```python
+import os
+os.environ.setdefault("NUMBA_DISABLE_CUDA", "1")
+import spaces
+```
+
+### 2. Load models at module scope, `.to("cuda")` eagerly
+
+```python
+pipe = DiffusionPipeline.from_pretrained("...", torch_dtype=torch.bfloat16).to("cuda")
+```
+
+Do **not** lazy-load inside `@spaces.GPU`. The hijack is designed for module-level placement; deferring it puts tens of seconds of checkpoint I/O + dtype cast + GPU move inside every cold request.
+
+Use the **string `"cuda"`** — never an integer device id. ZeroGPU re-allocates device ids per request, so `.to(0)`, `device_map={"": 0}`, `torch.cuda.set_device(0)` silently break.
+
+For plain `from_pretrained` loads, use `.to("cuda")`, **not** `device_map="cuda"` (which routes through `accelerate.set_module_tensor_to_device` and calls `torch._C._cuda_init()` at load time, bypassing the hijack). The exception is loaders that are ZeroGPU-aware — notably the `bitsandbytes` quantization path; `from_pretrained(..., quantization_config=BitsAndBytesConfig(...))` works with `device_map="cuda"`.
+
+**Preloading multiple variants** (e.g. base + refiner, image + video model) is fine as long as their combined VRAM fits. Load all of them sequentially at module scope into a dict, then key per request. Don't unload/reload between requests — that puts the load cost back on the user.
+
+### 3. Decorate the function Gradio binds
+
+ZeroGPU's startup scan walks Gradio's registered event handlers for `@spaces.GPU`-marked functions. If you decorate `inner_helper` but `click(fn=outer)` is what's wired up, you get `RuntimeError: No @spaces.GPU function detected during startup`. Always decorate the function passed to the event handler.
+
+```python
+@spaces.GPU(duration=60)
+def generate(prompt):
+    return pipe(prompt).images[0]
+
+btn.click(fn=generate, inputs=prompt_box, outputs=image_out)
+```
+
+## Sizing duration
+
+`@spaces.GPU(duration=N)` means "reserve N seconds of GPU time." Two failure modes:
+
+- **`ZeroGPU illegal duration`** — `N` exceeds the visitor's tier cap. Lowering `duration` is the only fix.
+- **`ZeroGPU quota exceeded`** — the visitor's remaining quota is less than `requested`. Compared as `requested vs remaining`, not `actual vs remaining` — so a 10-second task left at the default 60 s blocks the user as soon as their remaining drops below 60 s.
+
+Smaller `duration` also ranks **higher** in the queue. Both reasons push toward declaring the realistic worst case, not a comfortable margin.
+
+**Pick the value — don't guess.** A too-high duration deploys cleanly then errors on the first call; too-low silently truncates. Methodology:
+
+1. Ship with a placeholder (e.g. 180 s).
+2. Instrument with `time.perf_counter()` and return the seconds in the response.
+3. Run 2–3 representative calls via `gradio_client`.
+4. Set `duration = round(measured_max × 1.4)`.
+
+For input-dependent runtime, pass a **callable**:
+
+```python
+def _estimate(prompt, steps, *args, **kwargs):
+    # Swallow extras with *args, **kwargs — Gradio passes progress= positionally
+    # and a strict signature will raise "takes 5 positional arguments but 6 were given"
+    return min(240, 60 + int(steps * 3.5))
+
+@spaces.GPU(duration=_estimate)
+def generate(prompt, steps, ..., progress=gr.Progress(track_tqdm=True)):
+    ...
+```
+
+## Sizing memory: `large` vs `xlarge`
+
+`size="large"` (default) is half the backing card (48 GB on Blackwell). `size="xlarge"` is the full card (96 GB) and costs **2× quota** per second — plus higher queue waits. Use `large` unless the workload genuinely OOMs.
+
+Rough VRAM sizing:
+
+| Mode | Memory rule | 7B | 27B | 70B |
+|------|------------|------|------|------|
+| bf16 | `params × 2` GB | 14 GB ✓ large | 54 GB → xlarge | 140 GB → quant + xlarge |
+| int8 | `params × 1` GB | 7 GB ✓ large | 27 GB ✓ large | 70 GB → xlarge |
+| 4-bit (NF4 / int4) | `params × ~0.55` GB | 4 GB ✓ large | 15 GB ✓ large | 40 GB ✓ large |
+
+Numbers are for weights only; activations and KV cache add on top (significant for long context).
+
+## Quantization
+
+ZeroGPU supports two quantization stacks: **`bitsandbytes`** (drop-in for transformers, well-trodden) and **`torchao`** (torch-native, newer, smaller install). Pick by what your model's `from_pretrained` actually wires up; if both work, default to `bitsandbytes` for transformers LLMs and `torchao` for diffusers.
+
+### bitsandbytes (NF4 / int8)
+
+```python
+import spaces, torch
+from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
+
+bnb = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_quant_type="nf4",
+    bnb_4bit_use_double_quant=True,
+    bnb_4bit_compute_dtype=torch.bfloat16,
+)
+tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
+model = AutoModelForCausalLM.from_pretrained(
+    MODEL_ID,
+    quantization_config=bnb,
+    device_map="cuda",            # OK here — bnb's loader is ZeroGPU-aware
+    dtype=torch.bfloat16,
+).eval()
+```
+
+This is the one case where `device_map="cuda"` is **safe** on ZeroGPU at module scope (bitsandbytes' loader path intercepts cleanly). For non-bnb loads, stick to `.to("cuda")`.
+
+`load_in_8bit=True` swaps the 4-bit block for int8 — same hijack-safe loader. Bigger but higher quality, no `compute_dtype` knob.
+
+### torchao
+
+```python
+import spaces, torch
+from diffusers import DiffusionPipeline
+from torchao.quantization import quantize_, Int8WeightOnlyConfig
+
+pipe = DiffusionPipeline.from_pretrained(MODEL_ID, torch_dtype=torch.bfloat16).to("cuda")
+quantize_(pipe.transformer, Int8WeightOnlyConfig())   # mutates in place
+```
+
+`torchao` is more flexible (fine-grained per-module quantization, `Int4WeightOnlyConfig`, `Float8WeightOnlyConfig`, etc.) and works with diffusers' `from_pretrained(..., quantization_config=TorchAoConfig(...))` integration too. No CUDA build dependency — installs as a wheel.
+
+### Attention backends
+
+**Default to `attn_implementation="sdpa"`** — torch-native, works everywhere on sm_120, and is the right choice for the overwhelming majority of Spaces. Reach for a Flash-Attention backend only when the **upstream repo already references FA** (its config/model code defaults to or recommends `flash_attn`) — match what it expects rather than forcing SDPA. If FA then breaks on Blackwell, fall back to SDPA.
+
+**Flash Attention 2** — when you do need it, use the prebuilt wheel at `multimodalart/zerogpu-blackwell-wheels` ([`requirements.md`](requirements.md)), cp310–cp313. Drop the wheel URL in `requirements.txt`; no monkey-patch, no runtime build. The wheel's real `flash_attn_2_cuda` also satisfies xformers' import-time probes.
+
+**xformers** — same wheels dataset; auto-dispatch picks FA2 on sm_120 with no monkey-patch.
+
+**FA2 is the ceiling on ZeroGPU Blackwell — FA3 and FA4 cannot run on sm_120.** The RTX PRO 6000 (sm_120) lacks the TMEM / `tcgen05` tensor-memory subsystem the FA3/FA4 kernels are built on; those kernels only exist for Hopper (sm_90a) and datacenter Blackwell (sm_100a). This is a **hardware** limit, not a packaging gap — no wheel or branch fixes it:
+
+- `kernels-community/flash-attn3` (any revision, including `fake-ops-return-probs`), `vllm-flash-attn3`, `sgl-flash-attn3` all either fail to load (no matching build) or hard-fault at call time with `CUDA error: no kernel image is available for execution on the device` — which **kills the ZeroGPU worker** (surfaces as `GPU task aborted`); it does not fall back.
+- Verified empirically on the live runtime (`NVIDIA RTX PRO 6000 Blackwell … sm_120`, torch 2.11 / cu130) and confirmed upstream — every framework (vLLM, SGLang) falls back to FA2 on sm_120.
+
+So on ZeroGPU the ladder is **SDPA (default) → FA2 (only if the repo uses FA)**. Don't wire in FA3/FA4 — a stray `config.json` `kernels` entry loading `flash-attn3` will abort the worker on first GPU call.
+
+## Concurrency
+
+Handlers run **concurrently by default**. Three rules:
+
+1. **No mutable global state.** Handlers writing to a module-level dict / list race each other.
+2. **No fixed output paths.** Two concurrent calls writing to `output.png` clobber each other (and leak data across users). Use `tempfile.NamedTemporaryFile(suffix=...)`.
+3. **Read-only globals are safe** — models, tokenizers, configs loaded once and only read inside handlers.
+
+## Process isolation and pickle
+
+`@spaces.GPU` runs in a separate fork. Arguments and return values cross via pickle:
+
+- **Only picklable objects** in/out. File handles, locks, lambdas, closures over unpicklable state → `PicklingError`.
+- **Never return CUDA tensors.** Unpickling in the main process triggers `torch.cuda._lazy_init()`, which ZeroGPU blocks → the call hangs. Convert to CPU first: `return tensor.cpu()` or `.cpu().numpy()`.
+- CPU tensors, numpy arrays, PIL Images, plain Python objects work fine.
+- `gr.SelectData` is a special case — its `__getattr__` recurses under pickle. Extract the fields you need (`evt.index[0]`, etc.) in a thin un-decorated wrapper, pass plain values to the `@spaces.GPU` function.
+
+### `gr.State` across the fork
+
+`gr.State` is pickled on every yield. The handler receives a **copy**:
+
+- In-place mutations inside the fork are invisible to other handlers until you explicitly `yield` the mutated value back.
+- Yielding `gr.update()` for a state slot **skips** the update — other handlers continue to see pre-yield value.
+- For large state, minimize how often you yield it — ideally once at the end.
+- CUDA tensors inside state must be CPU-d before yielding (same `_lazy_init` issue).
+
+## Generators and streaming
+
+`@spaces.GPU` supports generator functions — first-class for progressive UI updates:
+
+```python
+@spaces.GPU(duration=120)
+def generate(prompt):
+    yield gr.update(value=None, label="Starting…")
+    for step in range(num_steps):
+        latent = step_fn(...)
+        yield gr.update(value=preview(latent), label=f"Step {step+1}/{num_steps}")
+    yield gr.update(value=final_image, label="Done")
+```
+
+`gr.Progress(track_tqdm=True)` and `yield` compete with each other — pick one.
+
+For streaming previews **inside** a diffusers `callback_on_step_end`, use a thread + queue inside the decorator (forks share threads):
+
+```python
+@spaces.GPU(duration=180)
+def generate(prompt, num_steps):
+    q = queue.Queue()
+    DONE = object()
+    def cb(pipe, step, t, kw):
+        q.put((step, taef1_preview(kw["latents"])))
+        return kw
+    def run():
+        out = pipeline(prompt=prompt, num_inference_steps=num_steps,
+                       callback_on_step_end=cb,
+                       callback_on_step_end_tensor_inputs=["latents"])
+        q.put((DONE, out))
+    threading.Thread(target=run, daemon=True).start()
+    while True:
+        idx, payload = q.get()
+        if idx is DONE: break
+        yield gr.update(value=payload, label=f"Step {idx+1}/{num_steps}")
+```
+
+**Do not** use `ProcessPoolExecutor` / `multiprocessing.Pool` inside `@spaces.GPU` — the daemonic fork can't spawn children (`AssertionError: daemonic processes are not allowed to have children`). Threads only.
+
+## Compilation (AoTI)
+
+`torch.compile` (JIT) is **not supported** on ZeroGPU — the forked GPU worker can't host the compile daemon. The supported path is PyTorch **ahead-of-time inductor (AoTI)**, wrapped by the `spaces` package (torch 2.8+, `spaces` ≥ 0.50). Its real value: compile a graph **once, offline**, publish it to the Hub, and have the serving Space load it — so serving cold starts pay no compile cost.
+
+### The `spaces` AoTI API
+
+- `spaces.aoti_capture(module)` — context manager. Run one real forward pass inside it; it patches `module.forward` to record that call's `args`/`kwargs` and abort the run. Gives you real example inputs for export.
+- `spaces.aoti_compile(exported, inductor_configs=None)` → an in-memory `ZeroGPUCompiledModel`.
+- `spaces.aoti_compile_and_save(package_dir, exported, inductor_configs=None)` — compile and write `{package_dir}/root/package.pt2`.
+- `spaces.aoti_apply(compiled, module)` — swap `module.forward` for an in-memory compiled model, same process.
+- `spaces.aoti_blocks_load(module, repo_id, variant=None)` — the high-level loader. For a module exposing `_repeated_blocks` (most diffusers transformers), it downloads `{BlockClass}[.{variant}]/package.pt2` from `repo_id` and patches every matching block. Weights stay **runtime inputs**, so one compiled block-graph serves any checkpoint with the same block architecture (e.g. a base and its distilled/turbo variant share one graph).
+
+### Recommended workflow: precompile the repeated block, load at runtime
+
+Split into a **compile Space** (run by hand, occasionally) and the **serving Space**, connected by a Hub **model** repo (`aoti_blocks_load` downloads with the default `repo_type="model"`).
+
+**Compile Space** — capture one repeated block's inputs, export, compile, save, upload:
+
+```python
+import os, spaces, torch, tempfile, shutil
+from pathlib import Path
+from huggingface_hub import HfApi
+
+pipe = DiffusionPipeline.from_pretrained(MODEL_ID, torch_dtype=torch.bfloat16).to("cuda")
+block = pipe.transformer.transformer_blocks[0]      # one representative repeated block
+BLOCK = type(block).__name__
+
+@spaces.GPU(duration=1500)
+def compile_and_upload():
+    with spaces.aoti_capture(block) as call:        # capture the block's real inputs…
+        pipe("a prompt", num_inference_steps=1)     # …from the first block call of a 1-step run
+    exported = torch.export.export(
+        block, call.args, call.kwargs,
+        dynamic_shapes=None,                        # start static; see note below
+        strict=False,
+    )
+    tmp = Path(tempfile.mkdtemp())
+    spaces.aoti_compile_and_save(tmp, exported)     # -> tmp/root/package.pt2
+    out = Path(tempfile.mkdtemp()); (out / BLOCK).mkdir()
+    shutil.copy(tmp / "root" / "package.pt2", out / BLOCK / "package.pt2")
+    HfApi(token=os.environ["HF_TOKEN"]).upload_folder(
+        folder_path=str(out), repo_id=REPO, repo_type="model")
+```
+
+The repo ends up as `{BlockClass}[.{variant}]/package.pt2`, optionally with a `config[.variant].json` listing custom `kernels` to fetch at load.
+
+**Serving Space** — one line at module scope, with an eager fallback:
+
+```python
+pipe = DiffusionPipeline.from_pretrained(MODEL_ID, torch_dtype=torch.bfloat16).to("cuda")
+try:
+    spaces.aoti_blocks_load(pipe.transformer, REPO)     # variant="fp8da" if you saved a variant
+except Exception as e:
+    print(f"AoTI load failed ({e!r}); running eager")
+```
+
+Only the repeated block is compiled, so the artifact is small and the same graph patches all N layers; compilation (minutes) happens offline, never on a serving cold start.
+
+### Footguns
+
+- **Construct the block identically in both Spaces.** The compiled graph's constant FQNs must line up with the serving module. If the block chooses kernels/norms via env var or import probe (e.g. a triton fused RMSNorm vs the pure-torch path), pin the *same* choice in compile and serving.
+- **Dynamic shapes.** `dynamic_shapes=None` bakes in the one resolution you captured. To serve variable sequence length / image size, pass `torch.export.Dim(...)` for those axes (e.g. `{"hidden_states": {1: torch.export.Dim("seq")}}`) — this is per-model tuning.
+- **Data-dependent inputs don't export.** A block taking per-sample python int lists (some double-stream blocks) can make `torch.export` refuse to make them dynamic — those stay eager. Single-stream blocks taking only packed tensors export cleanly.
+- **Custom kernels.** Ship a `config[.variant].json` with a `kernels` list (`repo_id` + `revision`); `aoti_blocks_load` calls `kernels.get_kernel(...)` before loading so the op is registered. (FA3 still won't run on sm_120 — see Attention backends.)
+- **Simpler in-process variant.** For a one-off, skip the Hub round-trip: `spaces.aoti_apply(spaces.aoti_compile(exported), module)` compiles and applies in the same process — but you re-compile on every cold start.
+
+### Reference Spaces
+
+- `multimodalart/Boogu-Image-0.1-Edit-aoti-compile` (compile + upload) and `multimodalart/Boogu-Image` (serving via block loading) — the blocks pattern end to end.
+- `zerogpu-aoti/Qwen-Image`, `zerogpu-aoti/Wan2` — artifact repos showing the `{BlockClass}.{variant}/package.pt2` + `config.{variant}.json` layout.
+- Deeper background on inductor configs and dynamic shapes: https://huggingface.co/blog/zerogpu-aoti
+
+## Local development
+
+**Do NOT** wrap `import spaces` in `try/except` with a no-op fallback. Off-ZeroGPU, the `spaces` package is *already* a true no-op — the heavyweight behavior is gated on `SPACES_ZERO_GPU=1`, set only on ZeroGPU. `@spaces.GPU` returns the undecorated function unchanged elsewhere. The Gradio base image installs `spaces` on every hardware tier, so a duplicate onto T4 / A10G / CPU works without code changes too.
+
+That said: **iterate ON the Space, not locally.** The Space environment (Python, torch, CUDA, drivers, env vars) differs from yours; passing local tests doesn't prove the Space works. Push early — even with the app not fully polished — and use the rung ladder ([`debugging.md`](debugging.md)) against the live URL.
+
+## Allocator config for memory pressure
+
+If your workload hits transient allocation spikes (high-res pixel-space ops, large attention activations, SR models, video DiTs) and you see:
+
+```
+RuntimeError: NVML_SUCCESS == r INTERNAL ASSERT FAILED at .../CUDACachingAllocator.cpp
+```
+
+set expandable segments at the **very top** of `app.py`, before any torch import:
+
+```python
+import os
+os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
+import spaces
+import torch
+```
+
+Often single-line fix for what looks like an OOM. See [`known-errors.md`](known-errors.md).
+
+## Example caching
+
+`gr.Examples` defaults on ZeroGPU:
+
+- `cache_examples=True`
+- `cache_mode="lazy"` (eager would pre-run examples at startup, but no GPU is attached at startup)
+
+Don't override to `cache_mode="eager"` on ZeroGPU — it will fail or burn the creator's daily quota. The cache is keyed by example **file path**, not content hash: regenerating an asset in place serves the stale cached output. Bump a `cache_version` constant if you replace example files.
+
+## Real-time sessions
+
+For real-time apps (webcam, audio streaming), the per-call fork model is too costly. ZeroGPU supports reusable "real-time sessions" — one GPU allocation amortized across many small requests. Reference Spaces:
+
+- https://huggingface.co/spaces/diffusers/unofficial-SDXL-Turbo-i2i-t2i
+- https://huggingface.co/spaces/huggingface-projects/rf-detr-realtime-webcam
+
+## When things go wrong
+
+For specific error strings (CUDA init order, illegal duration, allocator asserts, PicklingError, returning CUDA tensors, …): [`known-errors.md`](known-errors.md). It covers the ZeroGPU-specific patterns alongside everything else and is the single error lookup for the skill.
+
+When the log endpoint can't explain a failure (device-side asserts, OOM under specific shapes, race conditions in pickle), dev mode + SSH is the last-resort tool — see [`debugging.md`](debugging.md).

package/assets/hf-tooling/skills/huggingface-tool-builder/SKILL.md

@@ -0,0 +1,120 @@
+---
+name: huggingface-tool-builder
+description: Use this skill when the user wants to build tool/scripts or achieve a task where using data from the Hugging Face API would help. This is especially useful when chaining or combining API calls or the task will be repeated/automated. This Skill creates a reusable script to fetch, enrich or process data.
+---
+
+# Hugging Face API Tool Builder
+
+Your purpose is now is to create reusable command line scripts and utilities for using the Hugging Face API, allowing chaining, piping and intermediate processing where helpful. You can access the API directly, as well as use the `hf` command line tool. Model and Dataset cards can be accessed from repositories directly.
+
+## Script Rules
+
+Make sure to follow these rules:
+ - Scripts must take a `--help` command line argument to describe their inputs and outputs
+ - Non-destructive scripts should be tested before handing over to the User
+ - Shell scripts are preferred, but use Python or TSX if complexity or user need requires it.
+ - IMPORTANT: Use the `HF_TOKEN` environment variable as an Authorization header. For example: `curl -H "Authorization: Bearer ${HF_TOKEN}" https://huggingface.co/api/`. This provides higher rate limits and appropriate authorization for data access.
+ - Investigate the shape of the API results before commiting to a final design; make use of piping and chaining where composability would be an advantage - prefer simple solutions where possible.
+ - Share usage examples once complete.
+
+Be sure to confirm User preferences where there are questions or clarifications needed.
+
+## Sample Scripts
+
+Paths below are relative to this skill directory.
+
+Reference examples:
+- `references/hf_model_papers_auth.sh` — uses `HF_TOKEN` automatically and chains trending → model metadata → model card parsing with fallbacks; it demonstrates multi-step API usage plus auth hygiene for gated/private content.
+- `references/find_models_by_paper.sh` — optional `HF_TOKEN` usage via `--token`, consistent authenticated search, and a retry path when arXiv-prefixed searches are too narrow; it shows resilient query strategy and clear user-facing help.
+- `references/hf_model_card_frontmatter.sh` — uses the `hf` CLI to download model cards, extracts YAML frontmatter, and emits NDJSON summaries (license, pipeline tag, tags, gated prompt flag) for easy filtering.
+
+Baseline examples (ultra-simple, minimal logic, raw JSON output with `HF_TOKEN` header):
+- `references/baseline_hf_api.sh` — bash
+- `references/baseline_hf_api.py` — python
+- `references/baseline_hf_api.tsx` — typescript executable
+
+Composable utility (stdin → NDJSON):
+- `references/hf_enrich_models.sh` — reads model IDs from stdin, fetches metadata per ID, emits one JSON object per line for streaming pipelines.
+
+Composability through piping (shell-friendly JSON output):
+- `references/baseline_hf_api.sh 25 | jq -r '.[].id' | references/hf_enrich_models.sh | jq -s 'sort_by(.downloads) | reverse | .[:10]'`
+- `references/baseline_hf_api.sh 50 | jq '[.[] | {id, downloads}] | sort_by(.downloads) | reverse | .[:10]'`
+- `printf '%s\n' openai/gpt-oss-120b meta-llama/Meta-Llama-3.1-8B | references/hf_model_card_frontmatter.sh | jq -s 'map({id, license, has_extra_gated_prompt})'`
+
+## High Level Endpoints
+
+The following are the main API endpoints available at `https://huggingface.co`
+
+```
+/api/datasets
+/api/models
+/api/spaces
+/api/collections
+/api/daily_papers
+/api/notifications
+/api/settings
+/api/whoami-v2
+/api/trending
+/oauth/userinfo
+```
+
+## Accessing the API
+
+The API is documented with the OpenAPI standard at `https://huggingface.co/.well-known/openapi.json`.
+
+**IMPORTANT:** DO NOT ATTEMPT to read `https://huggingface.co/.well-known/openapi.json` directly as it is too large to process. 
+
+**IMPORTANT** Use `jq` to query and extract relevant parts. For example, 
+
+ Command to Get All 160 Endpoints
+
+```bash
+curl -s "https://huggingface.co/.well-known/openapi.json" | jq '.paths | keys | sort'
+```
+
+Model Search Endpoint Details
+
+```bash
+curl -s "https://huggingface.co/.well-known/openapi.json" | jq '.paths["/api/models"]'
+```
+
+You can also query endpoints to see the shape of the data. When doing so constrain results to low numbers to make them easy to process, yet representative.
+
+## Using the HF command line tool
+
+The `hf` command line tool gives you further access to Hugging Face repository content and infrastructure. 
+
+```bash
+❯ hf --help
+Usage: hf [OPTIONS] COMMAND [ARGS]...
+
+  Hugging Face Hub CLI
+
+Options:
+  --help                Show this message and exit.
+
+Commands:
+  auth                 Manage authentication (login, logout, etc.).
+  buckets              Commands to interact with buckets.
+  cache                Manage local cache directory.
+  collections          Interact with collections on the Hub.
+  datasets             Interact with datasets on the Hub.
+  discussions          Manage discussions and pull requests on the Hub.
+  download             Download files from the Hub.
+  endpoints            Manage Hugging Face Inference Endpoints.
+  env                  Print information about the environment.
+  extensions           Manage hf CLI extensions.
+  jobs                 Run and manage Jobs on the Hub.
+  models               Interact with models on the Hub.
+  papers               Interact with papers on the Hub.
+  repos                Manage repos on the Hub.
+  skills               Manage skills for AI assistants.
+  spaces               Interact with spaces on the Hub.
+  sync                 Sync files between local directory and a bucket.
+  upload               Upload a file or a folder to the Hub.
+  upload-large-folder  Upload a large folder to the Hub.
+  version              Print information about the hf version.
+  webhooks             Manage webhooks on the Hub.
+```
+
+The `hf` CLI command has replaced the now deprecated `huggingface-cli` command.