mlclaw 0.1.0 → 0.2.1

package/assets/hf-tooling/skills/huggingface-spaces/SKILL.md

@@ -0,0 +1,239 @@
+---
+name: huggingface-spaces
+description: Build, deploy, and maintain applications on Hugging Face Spaces — Gradio / Docker / Static SDKs, ZeroGPU and dedicated hardware, model loading, debugging, buckets, inference providers, community grants. Use whenever the user asks to create or host an app on Hugging Face, port code onto ZeroGPU, fix a Space that won't build or run, or otherwise work with `hf spaces …`, `@spaces.GPU`, Space README frontmatter, or the `spaces` Python package.
+---
+
+# Hugging Face Spaces
+
+Hugging Face Spaces host machine-learning applications. There are 1M+ today; each Space is a git repo. This skill covers creating, building, debugging, and maintaining them.
+
+## 0. Getting ready
+
+Before anything else:
+
+1. Check the `hf` CLI is installed: `which hf`. If not, `pip install -U huggingface_hub`.
+2. Check the user is logged in: `hf auth whoami`. If not, ask them to run `! hf auth login` in this session — they'll need a write-scoped token from https://huggingface.co/settings/tokens.
+3. Note `whoami`'s `canPay` and `isPro` flags — they gate hardware choices below.
+
+The `hf-cli` skill teaches an agent every `hf` command and is the recommended companion to this one. Install it with `hf skills add hf-cli` (add `--claude --global` to install for Claude Code as well, user-level).
+
+## 1. What a Space is
+
+A Space is a git repo with three possible SDKs:
+
+- **Gradio** — most Spaces. Python, fast iteration, supports ZeroGPU.
+- **Docker** — arbitrary container. Use when you need a non-Python stack or a pre-built template (Streamlit, Argilla, Shiny, etc. — full list at https://huggingface.co/docs/hub/spaces-sdks-docker). Does **not** support ZeroGPU.
+- **Static** — plain HTML, or a React/Svelte/Vue project built at deploy time. Use for in-browser ML (transformers.js / WebGPU / WebAssembly / onnxruntime-web), project pages, interactive reports, or Spaces that orchestrate other Spaces. No hardware needed.
+
+### Hardware tiers
+
+Free, no creator cost: **`cpu-basic`** and **`zero-a10g`** (ZeroGPU). Static Spaces are also free and don't need hardware.
+
+**`cpu-basic`** — 2 vCPU / 16 GB. For data viz, API-proxy Spaces, small CPU-bound models.
+
+**ZeroGPU (`zero-a10g`)** — dynamic, per-request GPU allocation on NVIDIA RTX PRO 6000 Blackwell (sm_120). Two sizes: `large` (half MIG, 48 GB, 1× quota) and `xlarge` (full, 96 GB, 2× quota). Free for the Space creator; Space visitors consume their own daily quota (~5 min free / 40 min Pro / 60 min Enterprise). **Gradio-only**, **PyTorch-first**. Requires the creator to be on a PRO / Team / Enterprise plan.
+
+**Dedicated GPU** (T4, L4, A10G, L40S, A100, H200) — billed to the Space creator by the hour. List + pricing: `hf spaces hardware`. Only the creator can attach these, and only if `canPay=True`. Use when ZeroGPU genuinely doesn't fit — non-PyTorch main model with heavy init, very-large-model long-context inference, etc.
+
+If a non-PRO user has a use case that wants ZeroGPU, you can still build it: create a `cpu-basic` Space, code the app for ZeroGPU, push, then request a community grant. See [`references/grants.md`](references/grants.md).
+
+For the authoritative reference: https://huggingface.co/docs/hub/spaces-overview
+
+## 2. Look for an existing demo first
+
+Before deciding how to build anything, search for prior art:
+
+```bash
+hf spaces search "<model name or task>" --sdk gradio --limit 10
+```
+
+If someone has built a similar Space, read its `app.py` and `requirements.txt` — that gives you the working pattern. Saves a lot of blind iteration. Mention to the user what you found before committing to an approach.
+
+## 3. Decide SDK and hardware
+
+Follow the user's explicit request first. If they were vague:
+
+- **Default for a public ML demo**: Gradio + ZeroGPU. Use this unless something below applies.
+- **The model's only inference path is non-PyTorch** (ONNX / TF / JAX / vLLM as the MAIN model, with heavy init): dedicated GPU.
+  - But: marginal non-torch tools (a small ONNX preprocessor, a TF utility) inside a torch-main pipeline are fine on ZeroGPU. The hijack only patches torch; init the non-torch lib inside `@spaces.GPU` and pay the short per-call init cost.
+- **Tiny / CPU-bound model, or API-proxy Space**: `cpu-basic` (`hardware`-free isn't applicable to Gradio).
+- **Browser-side ML or project page**: Static.
+- **Container with non-Python stack**: Docker.
+
+### Sourcing the model
+
+- **GitHub repo** — clone locally to read structure. If it already has a Gradio demo, the minimal viable path is to adapt it onto ZeroGPU (see [`references/zerogpu.md`](references/zerogpu.md)). Otherwise: read the README + inference code, prefer the PyTorch path, estimate VRAM (bf16 ≈ `params_B × 2` GB; 48 GB fits ≤24B params at bf16, or much larger with quantization — see [`references/zerogpu.md`](references/zerogpu.md) for quantization on ZeroGPU).
+- **HF model repo** — read its README, follow any linked GitHub.
+- **Paper / blog post** — look for an official or unofficial implementation. Don't reimplement unless trivial or the user explicitly asks.
+- **Vague request** — search Spaces first; surface results.
+
+If the model genuinely won't fit, check **Inference Providers** as an alternative: see [`references/inference-providers.md`](references/inference-providers.md). This avoids hosting the model at all.
+
+## 4. Create the Space
+
+```bash
+hf repos create <namespace>/<name> --type space --space-sdk <gradio|docker|static> \
+    [--flavor zero-a10g|cpu-basic|<paid-flavor>] \
+    [--secrets KEY=val] [--env KEY=val] \
+    --public|--private|--protected \
+    --exist-ok
+```
+
+- `--space-sdk` is required.
+- `--flavor` selects hardware. `zero-a10g` is the (legacy) identifier for ZeroGPU. Omit for `cpu-basic`. Run `hf spaces hardware` for the full paid list and pricing.
+- Visibility: `--public` (anyone can view), `--private` (only you), `--protected` (app is reachable but git repo / Files tab is private).
+- `--secrets KEY=val` becomes an environment variable inside the Space and is **not** visible to visitors. Use for API keys, gated-repo tokens (`HF_TOKEN=hf_…`), etc. Can also be set later via `hf spaces secrets set <id> KEY=val`.
+- `--env KEY=val` is **visible to visitors** — use only for non-sensitive config (`GRADIO_SSR_MODE=false`, `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`, etc.).
+
+> Note: `hardware:` in the README YAML is silently ignored — hardware is only set via `--flavor` at creation, or later via `hf spaces settings <id> --hardware <name>`.
+
+## 5. Build the app
+
+The Space now exists at `https://huggingface.co/spaces/<namespace>/<name>` but is empty.
+
+### README.md frontmatter
+
+Always required:
+
+```yaml
+---
+title: ...
+emoji: 🚀                # pick something representative
+colorFrom: blue          # red|yellow|green|blue|indigo|purple|pink|gray (only these)
+colorTo: indigo
+sdk: gradio              # gradio | docker | static
+sdk_version: 6.15.1      # latest stable unless you have a reason*
+app_file: app.py         # gradio only (docker / static use Dockerfile / index.html)
+short_description: ...   # ≤ 60 chars (server rejects longer)
+python_version: "3.12"   # ZeroGPU officially supports 3.10.13 and 3.12.12
+startup_duration_timeout: 30m   # default; bump to 1h for big LLMs / heavy downloads
+---
+```
+
+\* Default to the current latest stable, and **look up what that is** (`pip index versions gradio`, or the version a freshly-created Space defaults to) — the number above is a placeholder that goes stale, don't reuse it. Only pin older when the latest genuinely doesn't work for this Space: a custom component pins it, or you're adapting an existing demo and don't want to rewrite for 5.x→6.x breaking changes. If you need a 5.x, pick `5.50.0` (latest of the series; still supports custom components).
+
+All frontmatter options: https://huggingface.co/docs/hub/spaces-config-reference
+
+### Minimal ZeroGPU Gradio app
+
+```python
+import spaces           # MUST come before torch / diffusers / transformers
+import torch
+import gradio as gr
+from diffusers import DiffusionPipeline
+
+pipe = DiffusionPipeline.from_pretrained("<repo>", torch_dtype=torch.bfloat16).to("cuda")
+
+@spaces.GPU(duration=60)
+def generate(prompt: str):
+    """Generate an image from a text prompt."""   # docstring → API / MCP tool description
+    return pipe(prompt).images[0]
+
+gr.Interface(fn=generate, inputs=gr.Text(), outputs=gr.Image()).launch(mcp_server=True)
+```
+
+Three rules — full treatment in [`references/zerogpu.md`](references/zerogpu.md):
+
+1. **`import spaces` before torch / any CUDA-touching import.** It monkey-patches `torch.cuda.*`; once CUDA is initialized in the main process, it's too late.
+2. **Load the model at module scope, `.to("cuda")` eagerly.** ZeroGPU intercepts the call, packs weights to disk, and streams them into VRAM on the first `@spaces.GPU` entry. Lazy loading inside the decorator costs every user.
+3. **Decorate the function Gradio binds.** Estimate `duration` to the realistic worst case (smaller = higher queue priority and tighter quota check). For input-dependent runtime, pass a callable.
+
+### Examples, docstrings, and MCP
+
+- **Add `gr.Examples` whenever it makes sense** (the app takes input and representative inputs exist) — prefer the model/repo's own official examples. Keep example rows to the few inputs a user actually varies (prompt, image) and give the handler defaults for the rest (steps, seed, guidance) so a row is `["a prompt"]`, not a wall of knobs. Use `cache_examples=True, cache_mode="lazy"`. See [`references/gradio.md`](references/gradio.md).
+- **Give every API-triggered function a docstring and type hints.** Each Gradio event handler is exposed over the API; the docstring + signature are what a caller — and the MCP tool schema — sees.
+- **Launch with `demo.launch(mcp_server=True)`** (Gradio 5+) so the Space doubles as an MCP server: each API function becomes an MCP tool described by its docstring and hints.
+
+### requirements.txt
+
+Short version:
+
+- **Do NOT list**: `gradio`, `spaces`, `huggingface_hub` (preinstalled and platform-managed; pinning them causes resolution failures or silently breaks the ZeroGPU runtime).
+- **Do list if you use them**: `torchvision`, `torchaudio` (not preinstalled), plus everything else (`diffusers`, `transformers`, `accelerate`, `sentencepiece`, …).
+- ZeroGPU only accepts torch `2.8.0`, `2.9.1`, `2.10.0`, `2.11.0`. Default to leaving torch unpinned (the runtime preinstalls the latest). Only pin when a dep forces it.
+- For prebuilt CUDA-extension wheels (`flash_attn`, `xformers`, `pytorch3d`, `nvdiffrast`, `diff_gaussian_rasterization`, `torchmcubes`): use the prebuilt Blackwell wheels at `https://huggingface.co/datasets/multimodalart/zerogpu-blackwell-wheels/tree/main/wheels`. Full mapping + caveats in [`references/requirements.md`](references/requirements.md).
+
+### Per-SDK depth
+
+- **Gradio patterns** (themes, `gr.Examples`, streaming, custom HTML components, `gr.Server`): [`references/gradio.md`](references/gradio.md).
+- **Docker**: https://huggingface.co/docs/hub/spaces-sdks-docker. Examples: `hf spaces list --filter docker`.
+- **Static**: https://huggingface.co/docs/hub/spaces-sdks-static. For built SPAs, set `app_build_command: npm run build` and `app_file: dist/index.html` in frontmatter.
+- **ZeroGPU specifics** (decorator semantics, sizing, AoTI, generators, concurrency, pickle / `gr.State` across the worker boundary): [`references/zerogpu.md`](references/zerogpu.md) — read this whenever the Space targets ZeroGPU.
+
+
+## 6. Iterate on the Space, not locally
+
+Try to build a release candidate from the user quest locally and push it — then use the live URL as your test loop. The Space environment is the only one that matters; do not try to test locally. `python3 -m py_compile app.py` is the maximum local check worth doing before pushing.
+
+Push files with `hf upload <namespace>/<name> . --repo-type space`. **`--repo-type space` is required** — `hf upload` defaults to a *model* repo and will otherwise upload to (and silently create) a model repo of the same name. Add `--exclude "**/__pycache__/**"` so local bytecode caches aren't committed into the Space.
+
+Once pushed, pick the cheapest update mechanism for each change — hot-reload for pure Python edits, `hf upload` for code-only files hot-reload can't touch, full rebuild only when `requirements.txt` / `Dockerfile` / README frontmatter actually changed. Full ladder + footguns (hot-reload poisoning factory reboot, runtime.sha lag, etc.) in [`references/debugging.md`](references/debugging.md).
+
+## 7. Verify
+
+Don't trust `RUNNING` alone — the app can be running but broken. Four steps, in order:
+
+**A. Alive?** Stage + hardware:
+```bash
+hf spaces info <ns>/<name> --expand runtime
+```
+
+**B. Logs clean post-boot?** Read the run log to confirm startup finished without warnings or silent fallbacks:
+```bash
+hf spaces logs <ns>/<name> --tail 200
+```
+Look for model-load completion, no import warnings, no "falling back to CPU" / dtype downgrade messages, no `RUNNING` masking a half-broken app.
+
+**C. API actually responds.** With logs still tailing in another terminal (`hf spaces logs <ns>/<name> --follow`), call the endpoint:
+```python
+from gradio_client import Client, handle_file
+import os
+c = Client("<ns>/<name>", token=os.environ["HF_TOKEN"], httpx_kwargs={"timeout": 600})
+print(c.view_api())                    # discover endpoints — don't guess
+result = c.predict(..., api_name="/generate")
+```
+
+**D. Sniff output AND logs.** HTTP 200 ≠ correct output. Check both:
+```python
+head = open(result, "rb").read(16)
+# glTF / \x89PNG / RIFF…WEBP / RIFF…WAVE / [4:8]==b"ftyp" → png/jpg/webp/wav/mp4
+```
+And look at the run log emitted during the call — silent fallbacks (model snapping to a different size, missing optional dep, dtype downgrade) only show up there.
+
+Full smoke-test patterns (streaming endpoints, OAuth-gated Spaces, `gr.Server` custom routes): [`references/debugging.md`](references/debugging.md).
+
+## 8. Permanent storage (buckets)
+
+Spaces are stateless — `/data` is wiped on restart. If the Space needs to persist user uploads, generations, logs, or interact with a long-lived store, mount a **bucket**:
+
+```bash
+hf buckets create <ns>/<bucket-name>                                          # --private optional
+hf spaces volumes set <ns>/<space> -v hf://buckets/<ns>/<bucket-name>:/data   # read-write at /data
+```
+
+Buckets are paid storage; check `canPay` and confirm with the user. Full patterns (read-fast / write-durable, public bucket URLs, model-cache anti-pattern): [`references/buckets.md`](references/buckets.md).
+
+## 9. When things break
+
+Order of operations:
+
+1. Read the logs: `hf spaces logs <id> --build --follow` (build error) or `hf spaces logs <id> --follow` (runtime error). Find the **first** error, not the last.
+2. Grep [`references/known-errors.md`](references/known-errors.md) for the error string. Check if this is a known issue before trying your own fix — most common ZeroGPU / Gradio / dependency errors have a 1–2 line fix there.
+3. Iterate using the cheapest rung from [`references/debugging.md`](references/debugging.md). The vast majority of issues resolve with log-reading + smoke-test loops; interactive dev mode + SSH is a heavy-hammer last resort.
+
+If you solve an error that wasn't in the known-errors list, suggest the user PR it back to this skill so future runs benefit.
+
+---
+
+## Reference index
+
+| When to read | File |
+|---|---|
+| **How ZeroGPU works** + correct patterns (decorator, sizing, pickle, generators, real-time, AoTI) | [`references/zerogpu.md`](references/zerogpu.md) |
+| **Iterate + debug**: logs, rung ladder, smoke testing (and dev mode + SSH as a last resort) | [`references/debugging.md`](references/debugging.md) |
+| **Error-string lookup** — the single place for all error symptoms (Spaces, ZeroGPU, Gradio, deps) | [`references/known-errors.md`](references/known-errors.md) |
+| Pinning deps, picking wheels, torch-family alignment | [`references/requirements.md`](references/requirements.md) |
+| `gr.Examples` (add when it makes sense), themes, custom HTML components, `gr.Server`, MCP server (`mcp_server=True`) | [`references/gradio.md`](references/gradio.md) |
+| Persistent storage, public bucket URLs | [`references/buckets.md`](references/buckets.md) |
+| Community grant requests (non-PRO needing ZeroGPU) | [`references/grants.md`](references/grants.md) |
+| Provider proxy (zero-VRAM big LLM via Cerebras / Fireworks / Together / etc.) | [`references/inference-providers.md`](references/inference-providers.md) |

package/assets/hf-tooling/skills/huggingface-spaces/references/buckets.md

@@ -0,0 +1,89 @@
+# Persistent storage with Buckets
+
+Spaces are stateless. All data is wiped on restart / rebuild. For state that must survive (user uploads, generations, dynamic feeds, logs, growing databases): mount an HF **Bucket** — S3-like object storage living at `hf://buckets/<ns>/<bucket>`.
+
+Buckets are paid (per-TB storage). Check `whoami.canPay` and confirm with the user before creating one. Pricing + free tier: https://huggingface.co/storage.
+
+Full docs: https://huggingface.co/docs/hub/storage-buckets.
+
+## Create + attach
+
+```bash
+hf buckets create <ns>/<bucket-name>                                # --private optional
+hf spaces volumes set <ns>/<space> -v hf://buckets/<ns>/<bucket-name>:/data
+```
+
+After this, writes to `/data/` in the Space are durable. Reads come from the bucket via the Xet storage backend.
+
+To make the bucket files publicly addressable: leave the bucket public. Public bucket files are served at `https://huggingface.co/buckets/<ns>/<bucket>/resolve/<path>` (HTTP 302 redirect to a signed CDN URL). The Space writes once and the public URL works forever — no streaming proxy needed.
+
+## Write-durable, read-fast pattern
+
+For a feed-style Space (e.g. a community jam where users save generations and browse a public timeline), don't re-scan disk on every request. Module-level disk scan → in-memory list → every write appends to both:
+
+```python
+import os, json, uuid
+from datetime import datetime, timezone
+
+BUCKET_ID = "<ns>/<bucket-name>"
+BUCKET_URL = f"https://huggingface.co/buckets/{BUCKET_ID}/resolve"
+_feed = []
+
+def _load_feed():
+    root = "/data/songs"
+    if not os.path.isdir(root):
+        return
+    for sid in os.listdir(root):
+        meta = f"{root}/{sid}/meta.json"
+        if os.path.isfile(meta):
+            _feed.append(json.load(open(meta)))
+    _feed.sort(key=lambda s: s["created_at"], reverse=True)
+
+_load_feed()                                       # one scan at startup
+
+@app.api(name="save", time_limit=60)
+def save(audio_bytes: bytes, title: str):
+    sid = uuid.uuid4().hex[:12]
+    d = f"/data/songs/{sid}"; os.makedirs(d, exist_ok=True)
+    open(f"{d}/audio.wav", "wb").write(audio_bytes)
+    meta = {"id": sid, "title": title,
+            "url": f"{BUCKET_URL}/songs/{sid}/audio.wav",
+            "created_at": datetime.now(timezone.utc).isoformat()}
+    json.dump(meta, open(f"{d}/meta.json", "w"))
+    _feed.insert(0, meta)                          # cache stays current — no re-scan
+    return meta
+
+@app.api(name="feed", concurrency_limit=10)
+def feed(): return _feed[:50]                      # zero disk I/O
+```
+
+Reference Space using this pattern: https://huggingface.co/spaces/victor/ace-step-jam
+
+## Anti-pattern: bucket as model-weights cache
+
+**Do NOT** `snapshot_download(..., local_dir="/data/weights")` and load checkpoints from there. Bucket I/O is S3-paced; reading a 22 GB `safetensors` from `/data` during `from_pretrained` stalls past any `@spaces.GPU` duration cap.
+
+For model weights, let HF Hub re-download to local container disk on each cold start. With `HF_HUB_ENABLE_HF_TRANSFER=1` (set in the runtime by default) this is fast — typically much faster than streaming the same bytes through bucket I/O at request time.
+
+Bucket I/O is fine for occasional metadata reads (the feed pattern above) or saving user information. It is *not* fine as the path your model loader streams gigabytes through every cold start.
+
+## Cache redirects
+
+`/home/user/.cache` is read-only on ZeroGPU. Redirect transient caches at the top of `app.py`, before any library import that uses them:
+
+```python
+import os
+os.environ.setdefault("HF_HOME", "/data/.cache/huggingface")   # or /tmp on non-bucket Spaces
+os.environ.setdefault("HF_MODULES_CACHE", "/tmp/hf_modules")
+os.environ.setdefault("MPLCONFIGDIR", "/tmp/matplotlib")
+```
+
+Missing redirections fail silently or at first matplotlib / transformers / diffusers import.
+
+## Write access from the Space
+
+The Space's `HF_TOKEN` secret needs write permission on the bucket. Set via Settings → Secrets in the Space UI, or `hf spaces secrets set <id> HF_TOKEN=<token>`.
+
+## Security note
+
+Public bucket files are publicly accessible forever at their resolve URL. **Don't write PII** to a public bucket. If you need durable but private storage (e.g. per-user history requiring an HF login), keep the bucket private and gate reads through your Space's own auth.

package/assets/hf-tooling/skills/huggingface-spaces/references/debugging.md

@@ -0,0 +1,236 @@
+# Debugging and iteration
+
+How to iterate cheaply, read logs, and smoke-test. Dev mode + SSH exists as a last resort (covered at the bottom).
+
+## The rung ladder
+
+Pick the cheapest update mechanism that fits the change. Going one rung too high wastes 30 s – 15 min per cycle.
+
+| Rung | When | Command | Cost |
+|---|---|---|---|
+| 1. Hot-reload | Pure Python edit on a Gradio Space (SDK 6.1+), **no new deps** | `hf spaces hot-reload <id> -f app.py` | seconds, no rebuild |
+| 2. `hf upload` | Code-only change hot-reload can't handle (`gr.Server`, Streamlit, Docker entrypoint, non-Python file) | `hf upload <id> . --repo-type space --include '<file>'` | 30–90 s app restart |
+| 3. Full rebuild | `requirements.txt`, `Dockerfile`, README frontmatter, or hardware change | `hf upload <id> . --repo-type space --exclude "**/__pycache__/**" && hf spaces logs <id> --build --follow` | 1–15 min |
+
+`hf upload` defaults to a **model** repo — always pass `--repo-type space`, or it uploads to (and silently creates) a model repo of the same name. Also `--exclude "**/__pycache__/**"` so local bytecode caches don't get committed into the Space.
+| 4. Factory reboot | Container in inconsistent state (broken pip env, etc.) | `hf spaces restart <id> --factory-reboot` | full rebuild + cold start |
+
+### How hot-reload works
+
+`hf spaces hot-reload <id> -f app.py` patches the running Python process in place via `jurigged` (vendored inside the `spaces` package), then commits the change to the repo with a hot-reload marker that tells the platform to skip its usual restart. Seconds, no rebuild.
+
+Applies cleanly to function-body changes and new top-level symbols. Does **not** rerun module-level imports or one-time init — the model was loaded when `app.py` first ran and jurigged won't re-execute that load. New pip deps, README frontmatter, `Dockerfile`, and hardware changes need a full rebuild (rung 3).
+
+Independent of dev mode. Marked experimental in the CLI. Requires Gradio SDK 6.1+.
+
+### Footguns
+
+- **Hot-reload poisons factory reboot.** A commit that's only a hot-reload leaves runtime metadata that's valid only while the hot process is alive. `--factory-reboot` on top of one can fail with `fatal: could not read Username for 'https://huggingface.co'`. Recovery: push any normal `hf upload` commit (even a one-line no-op) first, then restart.
+- **`runtime.sha` lags repo SHA on restart.** `hf upload` succeeds → repo updates → `hf spaces info` keeps reporting the *previous* commit's SHA under `runtime` for several minutes while the new container loads. Poll `runtime.sha`, not just `stage`, and don't issue another restart until it flips.
+- **Concurrent uploads or restart-while-uploading collide.** Wait for one to finish.
+- **"Let me try locally first"** for anything that depends on the Space's Python / torch / CUDA env. The Space environment is the only one that matters. `python3 -m py_compile app.py` is the maximum local check worth doing before pushing.
+
+## Reading logs
+
+```bash
+hf spaces info <id> --expand runtime           # stage at a glance
+hf spaces logs <id> --build --follow           # build log, live
+hf spaces logs <id> --follow                   # run log, live
+hf spaces logs <id> --build --tail 500         # bigger window — default is small
+```
+
+Find the **first** error in the build log, not the last. Cascading errors after the first are noise.
+
+State machine (terminal states in bold):
+
+```
+BUILDING → APP_STARTING → RUNNING
+                      ↘ RUNTIME_ERROR
+        ↘ BUILD_ERROR
+        ↘ CONFIG_ERROR
+```
+
+For stage-specific lookups, see [`known-errors.md`](known-errors.md).
+
+## Smoke-test patterns
+
+A Space isn't done until a `gradio_client` call against the live URL exercises the endpoint end-to-end. Four steps in order — keep `hf spaces logs <id> --follow` running in another terminal throughout, so any silent fallback (model snapping to a different size, missing optional dep, dtype downgrade) surfaces.
+
+### A. Alive?
+
+```bash
+hf spaces info <id> --expand runtime --format json \
+  | python3 -c "import json,sys; r=json.load(sys.stdin)['runtime']; \
+                print(r['stage'], r.get('hardware','?'))"
+# expect: RUNNING zero-a10g
+```
+
+If `requested_hardware` is `cpu-basic` when you wanted GPU, your `--flavor` was rejected silently. Fix with `hf spaces settings <id> --hardware zero-a10g`.
+
+### B. Logs clean post-boot?
+
+```bash
+hf spaces logs <id> --tail 200
+```
+
+Confirm the model finished loading, no import warnings, no "falling back to CPU" / dtype-downgrade messages, no failing health checks the platform forgave. Do this before calling the API — many silent failures (a config typo loading the wrong model, a missing optional dep, a one-time init that errored but didn't crash boot) are only visible here.
+
+### C. API actually functions?
+
+Default — sync `gr.Interface` / `gr.Blocks` / `gr.ChatInterface` / `gr.Server` `@app.api`:
+
+```python
+from gradio_client import Client, handle_file
+import os
+
+c = Client("<ns>/<name>", token=os.environ["HF_TOKEN"],
+           httpx_kwargs={"timeout": 600})   # ≥ @spaces.GPU duration + 60s
+
+print(c.view_api())                          # discover endpoints — don't guess api_name
+
+result = c.predict(
+    handle_file("test.png"),                 # file inputs need handle_file()
+    "short prompt",
+    api_name="/generate",                    # matches @app.api(name=...) or the function name
+)
+```
+
+**Streaming endpoints** (function uses `yield` or `TextIteratorStreamer`) — `.predict()` returns only the final value. Iterate chunks via `.submit()`:
+
+```python
+job = c.submit("short prompt", api_name="/chat")
+for chunk in job: print(chunk, end="")
+# or job.result() for the final value
+```
+
+**`gr.Server` custom `@app.get/post(...)` routes** don't appear in `view_api()`. Hit them with plain HTTP:
+
+```python
+import httpx
+r = httpx.post(f"https://<subdomain>.hf.space/your_route",
+               json={...}, timeout=600,
+               headers={"Authorization": f"Bearer {os.environ['HF_TOKEN']}"})
+```
+
+**OAuth-gated Spaces** (`hf_oauth: true` + `gr.LoginButton`) — anonymous `Client` can't authenticate. Test interactively after sign-in, or capture a session token and pass via `httpx_kwargs={"headers": {...}}`.
+
+**MCP server mode** (`launch(mcp_server=True)`) — different protocol. Use an MCP client.
+
+### D. Output bytes AND logs look right?
+
+HTTP 200 ≠ correct output. Sniff both the returned file and the run log emitted during the call.
+
+```python
+head = open(path, "rb").read(16)
+# b'glTF...'          → glb
+# b'\x89PNG'          → png
+# b'\xff\xd8'         → jpeg
+# b'RIFF...WEBP'      → webp
+# b'RIFF...WAVE'      → wav
+# head[4:8]==b'ftyp'  → mp4
+# b'ply\n'            → ply
+```
+
+For text: non-empty, not all `<think>...</think>` (thinking-model leak), length reasonable. For images: returned dimensions match what was requested (some models snap to nearest preset).
+
+Look at the tailed run log alongside — silent fallbacks (model snapping resolution, missing optional dep falling back to a slower path, dtype downgrade) only show up there.
+
+### What NOT to do
+
+- **Don't launch Playwright / headless browser** to verify backend logic. The Gradio UI calls the same API `gradio_client` does — one `predict` tests both.
+- **Don't build mock-mode + local-server harnesses** before pushing. Local-green ≠ Space-green.
+- **Don't smoke-test with full-budget inputs.** Smallest input that exercises the GPU code path — short prompt, small image, low step count. You're verifying wiring, not quality.
+
+## Iterating on the Space, not locally
+
+The Space env is the only one that matters: Python, torch, CUDA, file paths, env vars, gradio version, the `spaces` hijack all differ from your laptop.
+
+Workflow:
+
+1. Decide SDK + hardware. Write the smallest `app.py` / `Dockerfile` + `requirements.txt` + README frontmatter — just enough that the entry point loads.
+2. Push immediately. Don't build a Playwright / mock harness first.
+3. Once `RUNNING`: verify with `gradio_client` against the real Space. That's your test loop.
+4. Iterate via the cheapest rung.
+
+`python3 -m py_compile app.py` is the maximum local check worth doing.
+
+## Last resort: dev mode + SSH
+
+Use only when:
+
+- A failure is non-deterministic (device-side asserts, OOM under specific shapes, race conditions).
+- You need `CUDA_LAUNCH_BLOCKING=1` or `gdb` to localize a CUDA error.
+- You'd burn 4+ build cycles trying variations from outside.
+
+Reading logs + grepping [`known-errors.md`](known-errors.md) + a tight `gradio_client` smoke loop solves the vast majority of issues. Dev mode is a heavy hammer.
+
+### Prerequisites
+
+1. **PRO / Team / Enterprise plan** — dev mode is a paid feature.
+2. **An SSH key registered on the user's HF profile.** Without this, SSH refuses the connection. If the user doesn't have one yet, they need to:
+   - Generate a keypair locally: `ssh-keygen -t ed25519 -f ~/.ssh/hf_dev -N ''` (no passphrase keeps automation simple; user can pick differently if they prefer).
+   - Add the **public** key (`~/.ssh/hf_dev.pub`) at https://huggingface.co/settings/keys.
+   - Keep the **private** key (`~/.ssh/hf_dev`) on the machine they'll SSH from.
+3. **The Space must be in `RUNNING` or `RUNTIME_ERROR`** before dev mode lets you in — not `BUILD_ERROR`. If it's in build error, push a stub `app.py` that boots cleanly first (e.g. `import gradio as gr; gr.Interface(lambda: 'ok', None, 'text').launch()`), then enable dev mode.
+
+### Enable
+
+No `huggingface_hub` Python wrapper yet — use the REST endpoint:
+
+```bash
+curl -s -X POST \
+  -H "Authorization: Bearer $HF_TOKEN" \
+  -H "Content-Type: application/json" \
+  "https://huggingface.co/api/spaces/<ns>/<space>/dev-mode" \
+  -d '{"enabled": true}'
+```
+
+### SSH in
+
+```bash
+ssh -i ~/.ssh/hf_dev -o BatchMode=yes -o StrictHostKeyChecking=accept-new \
+    <ns>-<space>@ssh.hf.space
+```
+
+Username is `<namespace>-<space>` **lowercase**, with `-` replacing `/`. Dots in the Space name become hyphens too.
+
+### Inside the VM
+
+It's a normal container at `/home/user/app/`. You can edit files, `pip install`, run repros, call `@spaces.GPU`-decorated functions interactively (they get a real GPU window).
+
+**`nvidia-smi` in the bare terminal will fail** with `NVML: Unknown Error`. Expected — ZeroGPU only exposes the real GPU inside `@spaces.GPU` calls. Don't assume the GPU is broken.
+
+**Edits in `/home/user/app/` don't survive** a restart, sleep, or dev-mode-disable. Only commits persist.
+
+### Smoke-test the fix inside the container
+
+Before exiting dev mode, verify the fix actually works under a real GPU window:
+
+```bash
+cat > /home/user/app/_devtest.py <<'PY'
+import spaces, torch
+from app import predict   # or whatever your @spaces.GPU function is
+print(predict(<realistic-args>))
+PY
+python3 _devtest.py
+```
+
+### Persist + exit
+
+Commit + push from inside the container (`git config user.email / user.name` first; the HF git remote works). Then disable dev mode:
+
+```bash
+curl -s -X POST -H "Authorization: Bearer $HF_TOKEN" \
+  -H "Content-Type: application/json" \
+  "https://huggingface.co/api/spaces/<ns>/<space>/dev-mode" \
+  -d '{"enabled": false}'
+```
+
+**Factory-reboot** to apply the pushed state (in dev mode the Space won't rebuild on commits):
+
+```python
+from huggingface_hub import HfApi
+HfApi(token=HF_TOKEN).restart_space("<ns>/<space>", factory_reboot=True)
+```
+
+Then re-run the outside-the-container smoke test. Dev-mode success does **not** guarantee post-rebuild success — different image, different process tree.