@damiangilgonzalez/harness 1.3.7

package/skills/modal/SKILL.md

@@ -0,0 +1,290 @@
+---
+name: modal
+description: "Use when running Python or GPU workloads serverlessly on Modal — modal.App, inline container Images, gpu= on @app.function, Volumes for weight caching, Cron schedules, ASGI endpoints, modal run vs serve vs deploy. NOT managed prediction APIs with no container of your own (that is replicate); NOT SSH-able GPU boxes rented by the hour (that is runpod)."
+tags: [modal, serverless, gpu, python, ai-infra, deployment, cron, web-endpoint]
+recommends: [replicate, runpod, fastapi, docker, python, llm-pipeline]
+origin: risco
+---
+
+# Modal — serverless Python & GPU as decorators
+
+Modal runs your Python on remote containers without you ever writing a Dockerfile or a YAML
+file. The mental model: **infrastructure is declared inline as Python decorators.** A
+`modal.App` is the deployable unit; each `@app.function` runs in its own container built from
+a `modal.Image` you describe in code; you attach a GPU, a Volume, or a Secret as keyword
+arguments and the platform provisions, scales to zero, and tears down for you. There is no
+control plane to babysit — the source file *is* the infra.
+
+Pinned stack: **modal 1.4.3** (released 2026-05-18), Python **3.10–3.14** (`>=3.10,<3.15`).
+Install with `pip install modal` then `modal setup` to authenticate. Everything below uses
+the Modal 1.0+ API; several pre-1.0 forms were removed and are called out as Bad→Good.
+
+## Not this skill
+
+Modal owns the serverless-container-as-decorators surface and its CLI lifecycle; the *contents*
+of your function belong elsewhere.
+
+| The job | Goes to |
+|---|---|
+| Calling a **managed prediction API** with no container of your own | [`replicate`](../replicate/SKILL.md) / [`together-fireworks`](../together-fireworks/SKILL.md) / [`fal`](../fal/SKILL.md) |
+| Renting a **persistent, SSH-able GPU box** by the hour/week | [`runpod`](../runpod/SKILL.md) |
+| **FastAPI design** (routing, Pydantic, deps) independent of host | [`fastapi`](../fastapi/SKILL.md) |
+| Writing a **Dockerfile** for a registry / k8s / Compose | [`docker`](../docker/SKILL.md) |
+| General **Python language/runtime** questions | [`python`](../python/SKILL.md) |
+| **RAG / LLM pipeline** orchestration logic itself | [`llm-pipeline`](../llm-pipeline/SKILL.md) |
+
+## Decision: which entrypoint?
+
+| You want… | Use | Persists after exit? |
+|---|---|---|
+| Run a function once and exit (script, batch) | `modal run app.py` + `@app.local_entrypoint()` | No (ephemeral) |
+| Hot-reload dev loop for a web endpoint | `modal serve app.py` | No (dies on Ctrl-C) |
+| A persistent named deployment (prod, schedules, endpoints) | `modal deploy app.py` | Yes |
+| Fan out work across many containers | `.map()` / `.starmap()` / `.spawn()` inside an entrypoint | n/a |
+
+Rule: **schedules and live web endpoints require `modal deploy`.** `modal run` exits when the
+entrypoint returns, so a Cron defined under `modal run` never fires. `modal serve` is for the
+dev loop only — it watches your files and redeploys on save, but the app vanishes when you
+stop it.
+
+## The minimal app skeleton
+
+```python
+import modal
+
+app = modal.App("hello-modal")
+
+# The image is the container spec. Build it once, reuse across functions.
+image = modal.Image.debian_slim(python_version="3.12").uv_pip_install("requests")
+
+
+@app.function(image=image)
+def fetch(url: str) -> int:
+    import requests  # imported INSIDE the function: it lives in the remote image, not locally
+
+    return len(requests.get(url).content)
+
+
+@app.local_entrypoint()
+def main() -> None:
+    # Runs on your laptop; .remote() ships the call to a Modal container.
+    print(fetch.remote("https://modal.com"))
+```
+
+Run it: `modal run app.py`. **Bad** = wiring infra with argparse + a bash launcher + a
+hand-rolled Dockerfile. **Good** = the decorators above; the app, image, and scaling are all
+declared in the one file. Note the in-function import: dependencies you `uv_pip_install` exist
+in the *remote* image, so import them inside the function (or guard top-level imports), not at
+module top where your laptop would need them too.
+
+## Images — pin, layer, cache
+
+Build images by chaining methods on `modal.Image`. Rules, each with its why:
+
+1. **Prefer `.uv_pip_install(...)` over `.pip_install(...)`** — it resolves and installs with
+   `uv`, materially faster image builds.
+2. **Pin versions** — `.uv_pip_install("torch==2.5.1", "transformers==4.46.0")`. Unpinned
+   deps make builds non-reproducible and silently drift on rebuild.
+3. **Order layers stable→volatile** — system packages and big wheels first, your fast-changing
+   code last. Modal caches each layer; a change busts that layer and everything after it.
+4. **Add your own code with `.add_local_dir(...)` / `.add_local_python_source(...)`**, not by
+   pip-installing your repo. These are applied last so editing your source doesn't rebuild torch.
+5. **`.from_registry("...")`** when you need a specific base image; **`.apt_install("ffmpeg")`**
+   for system binaries; **`.run_commands(...)`** for arbitrary build steps.
+
+```python
+image = (
+    modal.Image.debian_slim(python_version="3.12")
+    .apt_install("ffmpeg")                                   # stable: rarely changes
+    .uv_pip_install("torch==2.5.1", "transformers==4.46.0")  # heavy wheels, pinned
+    .add_local_python_source("my_pkg")                       # volatile: your code, applied last
+)
+```
+
+→ [`references/images-gpu-cookbook.md`](references/images-gpu-cookbook.md) for vLLM / torch+CUDA
+/ diffusers recipes and the download-once weight-cache pattern.
+
+## GPU — it's a string now
+
+In Modal 1.0+ the GPU is a **string** on the decorator. The old `modal.gpu.H100()` objects
+were removed.
+
+- Single GPU: `gpu="H100"`.
+- Count via colon: `gpu="A100:2"` (two A100s in one container).
+- Memory variant: `gpu="A100-80GB"` (also `A100-40GB`).
+- Fallback list (first available wins): `gpu=["H100", "A100", "any"]`.
+- Supported types: `T4, L4, A10, L40S, A100(-40GB/-80GB), RTX-PRO-6000, H100, H200, B200`.
+
+```python
+# Bad — removed API, raises at import.
+# @app.function(gpu=modal.gpu.A100())
+
+# Good — string form.
+@app.function(image=image, gpu="A100-80GB", timeout=600)
+def embed(texts: list[str]) -> list[list[float]]: ...
+```
+
+Pick the smallest GPU that fits: **T4/L4** for cheap inference and small models, **A10/L40S**
+mid-range, **A100/H100** for training and large-model serving, **H200/B200** for frontier-scale.
+GPU time is billed per second a container is alive — never attach a GPU to a CPU-only job, and
+keep `scaledown_window` tight so idle GPU containers don't burn money.
+
+## Scaling & lifecycle
+
+Tune these keyword args on `@app.function`, each with its why:
+
+| Param | Effect | Why |
+|---|---|---|
+| `min_containers=N` | Keep N warm instances always running | Kills cold starts for latency-sensitive endpoints (costs idle compute) |
+| `buffer_containers=N` | Pre-warm N extra beyond current load | Smooths bursty traffic |
+| `scaledown_window=300` | Seconds an idle container lingers before shutdown | Reuse hot containers across nearby calls; lower = cheaper, higher = warmer |
+| `timeout=600` | Max seconds a single call may run | Caps runaway jobs |
+| `retries=3` | Auto-retry failed inputs | Survives transient failures in `.map()` fan-outs |
+
+Migration note: `keep_warm` → **`min_containers`** and `container_idle_timeout` →
+**`scaledown_window`** in the 1.0 migration. The old names are gone.
+
+Concurrency within a container is now its own decorator: **`@modal.concurrent(max_inputs=N)`**
+stacked under `@app.function` (it replaces the old `allow_concurrent_inputs=` argument). Use it
+so one container handles N simultaneous requests instead of one-per-container.
+
+## Volumes & Secrets
+
+A `Volume` is a distributed filesystem you mount into containers to persist data across runs —
+the canonical use is caching downloaded model weights so cold starts skip the re-download.
+
+```python
+weights = modal.Volume.from_name("hf-cache", create_if_missing=True)
+
+
+@app.function(image=image, gpu="H100", volumes={"/cache": weights})
+def serve_model():
+    # Reader: refresh the view so you see writes from other containers.
+    weights.reload()
+    # ... load model from /cache ...
+
+
+@app.function(image=image, volumes={"/cache": weights})
+def download_weights():
+    # ... write files into /cache ...
+    weights.commit()  # WITHOUT this, writes are NOT durable across containers
+```
+
+**Gotcha:** writers must call `vol.commit()` to persist; readers call `vol.reload()` to see
+another container's committed writes. Forgetting `commit()` is the #1 "my cache is empty"
+bug — the files existed in that container and vanished with it.
+
+Secrets land as **environment variables** in the container:
+
+```python
+@app.function(image=image, secrets=[modal.Secret.from_name("hf-token")])
+def pull():
+    import os
+
+    token = os.environ["HF_TOKEN"]  # value injected from the named Modal Secret
+```
+
+Never bake a token into the image (`.run_commands("export TOKEN=...")`) — it's recorded in
+layer history. Use a Secret. The cookbook above also carries the HF/OpenAI secret patterns.
+
+## Web endpoints
+
+Stack a web decorator **under** `@app.function`. Pick by surface:
+
+| Decorator | Use for | Needs |
+|---|---|---|
+| `@modal.fastapi_endpoint()` | A single GET/POST function-as-URL | `fastapi[standard]` in image |
+| `@modal.asgi_app()` | A full FastAPI/Starlette app you return | `fastapi[standard]` |
+| `@modal.wsgi_app()` | A Flask/Django WSGI app | the framework |
+| `@modal.web_server(port=8000)` | Your own server process (e.g. vLLM) on a port | the server |
+
+**Decorator stack order matters:** `@app.function` is outermost (top), then optional
+`@modal.concurrent`, then the web decorator innermost (bottom, closest to `def`).
+
+```python
+@app.function(image=image, gpu="H100", min_containers=1, scaledown_window=300)
+@modal.concurrent(max_inputs=10)   # middle
+@modal.asgi_app()                  # innermost
+def web():
+    from fastapi import FastAPI
+
+    api = FastAPI()
+
+    @api.get("/health")
+    def health():
+        return {"ok": True}
+
+    return api
+```
+
+Develop with `modal serve app.py` (hot-reload); ship with `modal deploy app.py` (stable URL).
+For custom domains, proxy-auth tokens, batching (`@modal.batched`), and concurrency tuning →
+[`references/web-and-scaling.md`](references/web-and-scaling.md). For the FastAPI app's *own*
+design (routes, Pydantic, deps), that's [`fastapi`](../fastapi/SKILL.md) — this skill only mounts it.
+
+## Scheduled jobs
+
+```python
+# Fixed wall-clock time, with timezone — survives redeploys at the same clock time.
+@app.function(schedule=modal.Cron("0 6 * * *", timezone="America/New_York"))
+def nightly_report(): ...
+
+
+# Interval relative to deploy time.
+@app.function(schedule=modal.Period(hours=5))
+def every_five_hours(): ...
+```
+
+**Gotcha:** `Period` is measured from deploy time and **resets on every redeploy** — redeploy
+at 4:59 and your "every 5 hours" clock restarts. `Cron` is wall-clock stable; prefer it for
+"run at 6am" semantics. Either way you must **`modal deploy`** (not `modal run`) for the
+schedule to live on the platform.
+
+## Parallelism
+
+Fan a function out across containers without managing a pool:
+
+```python
+@app.local_entrypoint()
+def main():
+    urls = ["https://a.com", "https://b.com", "https://c.com"]
+    # .map: one arg per call, results in input order.
+    sizes = list(fetch.map(urls))
+    # .starmap: each item is an argument tuple. .spawn: fire-and-forget -> handle.get() later.
+    handle = fetch.spawn("https://slow.com")
+    print(sizes, handle.get())
+```
+
+`.map(iterable)` returns results in order by default; pass `order_outputs=False` to yield as
+they complete (faster when latencies vary). Combine with `retries=` on the function so a
+single bad input doesn't sink the batch.
+
+## Anti-patterns
+
+| Anti-pattern | Do instead |
+|---|---|
+| "I'll use `gpu=modal.gpu.A100()` like the old docs" | Removed in 1.0. Use the string `gpu="A100-80GB"`. |
+| "Attach a GPU, it might speed up this CPU job" | GPU is billed per second alive. CPU-only job → no `gpu=`. |
+| "My files are written, the Volume will keep them" | Not without `vol.commit()` (writer) / `vol.reload()` (reader). |
+| "Pin later — `uv_pip_install('torch')` is fine for now" | Unpinned deps drift; builds aren't reproducible. Pin every version. |
+| "`modal run` it, the endpoint/schedule will stay up" | `run` is ephemeral; it exits. Use `modal deploy` for anything persistent. |
+| "Order the decorators however — Modal figures it out" | `@app.function` outermost, web decorator innermost. Wrong order errors. |
+| "Bake the HF token into the image with `run_commands`" | Leaks into layer history. Use `modal.Secret.from_name(...)`. |
+| "Just call the model via a managed API through Modal" | If you write no container, that's a managed-API job → [`replicate`](../replicate/SKILL.md). |
+| "I need a box to SSH into for a week" | That's a persistent rental → [`runpod`](../runpod/SKILL.md), not Modal's scale-to-zero. |
+| "Set `min_containers` high so it's always fast" | Idle warm containers cost money 24/7. Tune `scaledown_window` first. |
+
+## Verify
+
+[`scripts/verify.sh`](scripts/verify.sh) `[TARGET]` statically lints the nearest emitted Modal
+`*.py`: it requires a `modal.App(`, **fails** if the removed `modal.gpu.` object form appears,
+checks that any web decorator sits under an `@app.function`, and that any `Volume` uses
+`from_name(..., create_if_missing=...)`. It runs `python -c "import modal"` only if modal is
+installed (skip-pass otherwise), needs **no Modal credentials**, and exits 0 on an empty target.
+
+## Project grounding (docs)
+
+In a project with a `docs/` layer (the [`harness`](../harness/SKILL.md) wiki), read
+`docs/wiki/stack/modal.md` first, then record this app's real Modal choices there — GPU types,
+image base, Volume names, schedule, endpoint shape — and index it in `docs/wiki/index.md`. No
+`docs/`? Skip silently.

package/skills/modal/evals/README.md

@@ -0,0 +1,29 @@
+# Eval harness — `modal` skill
+
+These cases are run by an **agent harness** (an LLM agent with the skill catalog available for
+routing), not a plain shell script. `cases.yaml` is the fixture; this file is the procedure.
+No Modal account or credentials are needed — every check is about *routing* and the *generated
+code rubric*, not live execution against Modal.
+
+## 1. Triggering accuracy
+
+Load only the skill descriptions (this `modal` one plus the bare names of routing siblings:
+replicate, runpod, fastapi, docker, python, llm-pipeline), not the bodies. For each prompt in
+`should_trigger` / `should_not_trigger`, ask which skill the agent would invoke, running 3–5
+trials per prompt. `should_trigger` passes if `modal` wins the majority; `should_not_trigger`
+passes if `modal` does **not** fire (bonus if it routes to the named `route_to` sibling). Pass
+bar: ≥90% of prompts, and no near-miss may fire `modal` in a majority of its trials. The
+Catalan/Spanish and the `gpu=modal.gpu.A100()` prompts are the ones to watch for flakiness.
+
+## 2. Capability uplift (with vs without)
+
+For each `capability` scenario, run the agent twice — once with no `modal` skill loaded, once
+with it fully loaded — and grade each output against the scenario's `must_include` checklist
+(a point counts only if genuinely correct, not merely mentioned). Pass bar: WITH-skill
+coverage ≥80% of the rubric, and WITH must clearly beat WITHOUT. The load-bearing points are
+the string GPU form, `Volume.from_name(create_if_missing=True)` with commit/reload, correct
+web-decorator stack order under `@app.function`, and `modal deploy` (not `run`) for anything
+persistent. Any rubric point the skill never produces is a gap to fix in `SKILL.md`.
+
+This is judgment-based, not byte-exact; record trial counts and the judge (human or model)
+alongside results for reproducibility.

package/skills/modal/evals/cases.yaml

@@ -0,0 +1,50 @@
+skill: modal
+
+should_trigger:
+  - prompt: "Run my Python embedding job on an H100 without managing any servers — just ship the function and have it scale."
+    why: "Core serverless GPU on Modal: @app.function + gpu string + scale-to-zero. The defining use case."
+  - prompt: "My @app.function fails with gpu=modal.gpu.A100(). What's the right way to ask for that GPU now?"
+    why: "Non-obvious 1.0-migration error: the modal.gpu.X() object form was removed for the string form. Modal-specific symptom."
+  - prompt: "Cache my HuggingFace model weights across runs so cold starts don't re-download 16GB every time."
+    why: "Volume-backed weight-cache pattern (from_name + commit/reload) — squarely Modal state, phrased without naming Modal."
+  - prompt: "modal serve isn't hot-reloading my fastapi_endpoint changes — edits to the handler don't show up."
+    why: "Non-obvious dev-loop symptom tied to Modal's serve-vs-deploy lifecycle and web decorators."
+  - prompt: "Schedule a nightly scrape in the cloud at 6am New York time and keep it running."
+    why: "modal.Cron with timezone + the deploy-not-run gotcha for schedules to live. Cron/Period territory."
+  - prompt: "Desplegar un model de difusió amb GPU serverless i exposar-lo per HTTP amb endpoint propi."
+    why: "Catalan: serverless GPU + web endpoint (web_server/asgi_app) — the GPU-serving-over-HTTP core."
+  - prompt: "Ejecutar Python en GPU serverless que escale a cero y solo pague por segundo de uso."
+    why: "Spanish: scale-to-zero per-second GPU compute — Modal's billing/scaling model, not a managed API."
+
+should_not_trigger:
+  - prompt: "Run Stable Diffusion through a hosted prediction API where I just POST a prompt and get an image — no infra code."
+    route_to: "replicate"
+    why: "Managed prediction endpoint with no container of your own; Modal is for code-first containers."
+  - prompt: "Rent a persistent GPU pod I can SSH into and keep running for a week to iterate by hand."
+    route_to: "runpod"
+    why: "Persistent long-lived rentable GPU box; Modal is scale-to-zero serverless, not per-pod rental."
+  - prompt: "Design my FastAPI routers, Pydantic v2 models, and dependency injection for a user API."
+    route_to: "fastapi"
+    why: "FastAPI app design independent of host; Modal only mounts such an app, it doesn't design it."
+  - prompt: "Write a Dockerfile and push the image to a registry to deploy on Kubernetes."
+    route_to: "docker"
+    why: "Generic container/registry/k8s mechanics; Modal builds images inline in Python, no Dockerfile."
+  - prompt: "Structure my RAG retrieval, chunking, and reranking pipeline logic."
+    route_to: "llm-pipeline"
+    why: "LLM/RAG orchestration logic itself; Modal would only host it, the pipeline design is the sibling's."
+
+capability:
+  - scenario: "User asks: 'Build a Modal app that serves a vLLM endpoint on an H100 with a warm pool and a weight-cache volume, and tell me how to ship it.'"
+    must_include:
+      - "modal.App(...) defined as the deployable unit"
+      - "modal.Image with .uv_pip_install/.pip_install including vllm and fastapi[standard] (or equivalent server deps)"
+      - "GPU requested as the string form gpu=\"H100\" (NOT modal.gpu.H100())"
+      - "modal.Volume.from_name(..., create_if_missing=True) mounted via volumes={\"/path\": vol} for weight caching, with commit/reload noted"
+      - "Warm pool + lifecycle: min_containers and scaledown_window (and timeout) on @app.function"
+      - "A web decorator (@modal.web_server(port=...) for vLLM's own server, or @modal.asgi_app()) stacked UNDER @app.function in correct order"
+      - "Ship with `modal deploy` (not `modal run`) because the endpoint must persist"
+  - scenario: "User asks: 'Schedule a Python function to run every night at 6am New York time on Modal, and make sure it actually keeps firing.'"
+    must_include:
+      - "schedule=modal.Cron(\"0 6 * * *\", timezone=\"America/New_York\") on @app.function (Cron preferred over Period for wall-clock time)"
+      - "Note that Period resets on every redeploy whereas Cron is wall-clock stable (if Period is mentioned at all)"
+      - "Must `modal deploy` (not `modal run`) for the schedule to live on the platform"

package/skills/modal/references/images-gpu-cookbook.md

@@ -0,0 +1,160 @@
+# Images, GPU & state cookbook
+
+Deep recipes for `modal.Image` builds, GPU selection, and Volume/Secret-backed state.
+All forms are Modal 1.0+ (modal 1.4.3). GPU is always a string; `modal.gpu.X()` is removed.
+
+## Image recipes
+
+### Torch + CUDA inference
+
+```python
+import modal
+
+image = (
+    modal.Image.debian_slim(python_version="3.12")
+    .uv_pip_install(
+        "torch==2.5.1",
+        "transformers==4.46.0",
+        "accelerate==1.1.1",
+    )
+)
+```
+
+`debian_slim` already ships a CUDA-capable userspace for the GPU you attach — you do not
+install CUDA yourself. Pin every wheel so a rebuild is byte-for-byte reproducible.
+
+### vLLM server (own HTTP server via web_server)
+
+```python
+vllm_image = (
+    modal.Image.debian_slim(python_version="3.12")
+    .uv_pip_install("vllm==0.6.4", "fastapi[standard]")
+)
+
+
+@app.function(image=vllm_image, gpu="H100", scaledown_window=300, timeout=600)
+@modal.concurrent(max_inputs=32)
+@modal.web_server(port=8000, startup_timeout=300)
+def serve():
+    import subprocess
+
+    # vLLM exposes its own OpenAI-compatible server on :8000; web_server proxies it.
+    subprocess.Popen(
+        "vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8000",
+        shell=True,
+    )
+```
+
+`web_server` is the right decorator when the process *is* the server (vLLM, TGI, a custom
+uvicorn). Use `startup_timeout` to allow weight loading before health checks begin.
+
+### Diffusers image generation
+
+```python
+diff_image = modal.Image.debian_slim(python_version="3.12").uv_pip_install(
+    "diffusers==0.31.0", "torch==2.5.1", "transformers==4.46.0", "accelerate==1.1.1"
+)
+```
+
+### System binaries (ffmpeg, git)
+
+```python
+image = (
+    modal.Image.debian_slim()
+    .apt_install("ffmpeg", "git")     # system layer first — rarely changes
+    .uv_pip_install("imageio-ffmpeg==0.5.1")
+)
+```
+
+### Custom base image
+
+```python
+image = modal.Image.from_registry(
+    "nvidia/cuda:12.4.0-devel-ubuntu22.04", add_python="3.12"
+).uv_pip_install("xformers==0.0.28.post3")
+```
+
+Reach for `from_registry` only when `debian_slim` can't express what you need (a specific CUDA
+devel toolchain, a vendor image). It's heavier and slower to pull.
+
+## GPU selection matrix
+
+| GPU string | Rough fit | Notes |
+|---|---|---|
+| `"T4"` | Cheap small-model inference, embeddings | 16GB; cheapest |
+| `"L4"` | Efficient inference, light vision | 24GB |
+| `"A10"` | Mid-range inference | 24GB |
+| `"L40S"` | Mid/large inference, some training | 48GB |
+| `"A100-40GB"` / `"A100-80GB"` | Training, 7–70B serving | colon for count: `"A100:4"` |
+| `"H100"` | Fast large-model training/serving | 80GB |
+| `"H200"` | Larger memory frontier work | 141GB |
+| `"B200"` | Frontier-scale | newest |
+| `"RTX-PRO-6000"` | Workstation-class | |
+
+Multi-GPU: `gpu="A100:8"`. Fallbacks for capacity: `gpu=["H100", "A100", "any"]` — Modal grabs
+the first available so a job isn't stuck waiting for one exact type. Always start with the
+smallest GPU your model fits in; step up only when you OOM.
+
+## Volume-backed weight cache (download once, mount everywhere)
+
+The pattern that makes GPU cold starts cheap: download weights into a Volume once, then mount
+the same Volume read-mostly into every serving container.
+
+```python
+import modal
+
+app = modal.App("cached-llm")
+cache = modal.Volume.from_name("hf-cache", create_if_missing=True)
+image = modal.Image.debian_slim(python_version="3.12").uv_pip_install(
+    "huggingface_hub==0.26.2", "transformers==4.46.0", "torch==2.5.1"
+)
+CACHE_DIR = "/cache"
+
+
+@app.function(
+    image=image,
+    volumes={CACHE_DIR: cache},
+    secrets=[modal.Secret.from_name("hf-token")],
+    timeout=1800,
+)
+def warm_cache(repo: str = "meta-llama/Llama-3.1-8B-Instruct"):
+    import os
+    from huggingface_hub import snapshot_download
+
+    snapshot_download(repo, cache_dir=CACHE_DIR, token=os.environ["HF_TOKEN"])
+    cache.commit()  # persist the download so serving containers can read it
+
+
+@app.function(image=image, gpu="H100", volumes={CACHE_DIR: cache})
+def generate(prompt: str):
+    cache.reload()  # see what warm_cache committed
+    import os
+
+    os.environ["HF_HOME"] = CACHE_DIR  # transformers reads weights from the mounted cache
+    # ... load model from CACHE_DIR, no re-download ...
+```
+
+Run `modal run app.py::warm_cache` once, then deploy the serving function. Without
+`cache.commit()` the download evaporates with the container; without `cache.reload()` the
+serving container may not see it yet.
+
+## Secret patterns
+
+```python
+# Reference an existing named Secret (created via dashboard or `modal secret create`).
+hf = modal.Secret.from_name("hf-token")       # injects HF_TOKEN env var
+openai = modal.Secret.from_name("openai")     # injects OPENAI_API_KEY
+
+# Ad-hoc inline (for local dev only; prefer named Secrets in prod).
+inline = modal.Secret.from_dict({"MY_FLAG": "1"})
+
+
+@app.function(secrets=[hf, openai])
+def call():
+    import os
+
+    return os.environ["OPENAI_API_KEY"][:6]
+```
+
+Secrets become environment variables inside the container. Never put them in the image, in
+`run_commands`, or in source — those persist in layer history and in git.

package/skills/modal/references/web-and-scaling.md

@@ -0,0 +1,138 @@
+# Web endpoints & scaling — deep dive
+
+The four web decorators, concurrency/batching, and the cold-start vs cost tradeoff.
+All Modal 1.0+ (modal 1.4.3). Stack order is always: `@app.function` outermost, then optional
+`@modal.concurrent`, then the web decorator innermost.
+
+## The four web decorators
+
+### `@modal.fastapi_endpoint()` — one function, one URL
+
+The lightest option: turn a single function into a GET/POST endpoint. Needs
+`fastapi[standard]` in the image.
+
+```python
+image = modal.Image.debian_slim().uv_pip_install("fastapi[standard]")
+
+
+@app.function(image=image)
+@modal.fastapi_endpoint(method="POST", docs=True)
+def predict(payload: dict) -> dict:
+    return {"echo": payload}
+```
+
+Pass `method=` for the verb and `docs=True` to expose the OpenAPI docs page. Good for a quick
+inference URL; reach for `asgi_app` when you need routing.
+
+### `@modal.asgi_app()` — a full FastAPI / Starlette app
+
+Return an ASGI app object. Use when you want multiple routes, middleware, or a real API surface.
+
+```python
+@app.function(image=image)
+@modal.concurrent(max_inputs=20)
+@modal.asgi_app()
+def api():
+    from fastapi import FastAPI
+
+    web = FastAPI()
+
+    @web.get("/items/{item_id}")
+    def read(item_id: int):
+        return {"item_id": item_id}
+
+    return web
+```
+
+The *design* of this FastAPI app (Pydantic models, dependencies, error envelopes) is the
+`fastapi` sibling's job — Modal only hosts it.
+
+### `@modal.wsgi_app()` — Flask / Django
+
+Return a WSGI callable. Same stacking; install the framework in the image.
+
+```python
+@app.function(image=modal.Image.debian_slim().uv_pip_install("flask==3.1.0"))
+@modal.wsgi_app()
+def flask_app():
+    from flask import Flask
+
+    web = Flask(__name__)
+
+    @web.route("/")
+    def index():
+        return "ok"
+
+    return web
+```
+
+### `@modal.web_server(port=...)` — run your own server
+
+When the workload spawns its own HTTP server (vLLM, TGI, a uvicorn process), use `web_server`
+and let Modal proxy the port.
+
+```python
+@app.function(image=vllm_image, gpu="H100")
+@modal.web_server(port=8000, startup_timeout=300)
+def serve():
+    import subprocess
+
+    subprocess.Popen("vllm serve <model> --port 8000", shell=True)
+```
+
+`startup_timeout` covers slow model loads before health checks begin.
+
+## Concurrency: `@modal.concurrent`
+
+One container, many simultaneous inputs. Replaces the removed `allow_concurrent_inputs=` arg.
+
+```python
+@app.function(image=image)
+@modal.concurrent(max_inputs=100)
+def handler(x): ...
+```
+
+Set `max_inputs` to what a single container can handle concurrently without resource
+contention — for I/O-bound work it can be high; for GPU inference it's bounded by VRAM and
+batch capacity. Too high → OOM / latency spikes; too low → you pay for more containers than
+needed.
+
+## Batching: `@modal.batched`
+
+Coalesce individual calls into batches server-side (great for GPU inference throughput).
+
+```python
+@app.function(image=image, gpu="A100")
+@modal.batched(max_batch_size=16, wait_ms=50)
+def embed(texts: list[str]) -> list[list[float]]:
+    # Modal collects up to 16 calls (or waits 50ms) and passes them as one list.
+    ...
+```
+
+`max_batch_size` caps the batch; `wait_ms` is how long to wait to fill it. Trade a little
+latency for much higher GPU utilization.
+
+## Scaling & cold-start tradeoff
+
+| Param | Lower value | Higher value |
+|---|---|---|
+| `min_containers` | Scale fully to zero — cheapest, cold starts on first hit | Warm pool always ready — no cold start, pays idle 24/7 |
+| `buffer_containers` | No pre-warm buffer | Absorbs traffic bursts before autoscale catches up |
+| `scaledown_window` | Containers die fast after idle — cheaper, more cold starts | Stay warm longer — fewer cold starts, more idle cost |
+
+Tuning order for a latency-sensitive endpoint: first widen `scaledown_window` so nearby
+requests reuse a hot container; only set `min_containers >= 1` if you truly need zero cold
+starts (e.g. a customer-facing API with an SLA). For a GPU endpoint, even `min_containers=1`
+is a real ongoing cost — measure before pinning it.
+
+## Custom domains, auth, proxy tokens
+
+- **Custom domain:** map your domain to the deployed app in the Modal dashboard; the
+  deployment URL is stable after `modal deploy`.
+- **Endpoint auth:** protect a `fastapi_endpoint`/`asgi_app` with Modal proxy auth tokens so
+  only callers with the token reach it — configure tokens on the function and pass them as
+  request headers. Prefer this over hand-rolling auth inside the handler for machine-to-machine
+  endpoints.
+- **Dev vs prod:** iterate with `modal serve` (hot reload, ephemeral URL); ship with
+  `modal deploy` (persistent named app, stable URL). Never rely on a `modal serve` URL in
+  production — it dies when you stop the process.