@dsalazardev/ai-hub-cli 0.0.0-stage → 0.1.0

package/skills/shared/replicate/SKILL.md

@@ -0,0 +1,210 @@
+---
+name: replicate
+description: "Use when running, packaging, deploying or scaling a model on the Replicate platform from code — blocking run versus async predictions, handling FileOutput, deployments with warm private endpoints and autoscaling, packaging with Cog, verifying webhook signatures, and cutting GPU spend. NOT crafting image prompts and parameters or picking image model families (that is `replicate-images`)."
+tags: [replicate, cog, deployments, webhooks, model-serving, gpu, ai-infra]
+recommends: [replicate-images, modal, runpod, huggingface, webhooks, cost-tracking, docker]
+origin: ai-hub
+---
+
+# Replicate platform operations
+
+This skill is about **how a model runs in production on Replicate** — clients, async, deployments,
+Cog packaging, webhooks, scaling, and spend. It is the platform-engineering counterpart to image
+prompt craft. If the question is *what prompt, aspect ratio, or model family produces a good image*,
+that is `replicate-images`, not this skill. Here the mental model is: **a prediction is a job. You
+either wait for it, poll it, or get pinged about it — and where it runs (shared cold pool vs a private
+warm deployment) is a cost-and-latency dial you set deliberately.**
+
+Pinned facts (verified 2026-06-02): Python client `replicate` 1.x (latest 1.0.7), Python 3.8+;
+JS client `replicate` on npm; auth via `REPLICATE_API_TOKEN`. A `2.0.0aN` alpha exists on PyPI but
+is NOT the default — pin `replicate>=1,<2` so a fresh install never silently pulls it.
+
+## Decision: how should this model run?
+
+Pick the row by latency tolerance and whether your process can block. Do not default to `run()` for
+everything — a 10-minute job inside a web request will time out and burn a worker.
+
+| Pattern | Latency | Blocks your process? | Cost shape | Use when |
+|---|---|---|---|---|
+| `replicate.run(...)` | seconds | Yes — waits to completion | per-prediction, shared pool | interactive/quick calls, scripts, CLIs |
+| `predictions.create()` + poll | minutes | Yes, but you control the loop | per-prediction, shared pool | long job, a worker can babysit it |
+| `predictions.create(webhook=...)` | minutes+ | No — fire and forget | per-prediction, shared pool | long job, the request must return now |
+| Deployment (private endpoint) | low + steady | depends on call style above | warm floor + per-prediction | sustained traffic, need warm/private/autoscale cap |
+
+Rule: if a human or HTTP request is waiting longer than a few seconds, do not block on `run()` —
+switch to predictions + webhook. Why: synchronous timeouts kill the request but the GPU job keeps
+running and billing.
+
+## Auth & install
+
+```bash
+export REPLICATE_API_TOKEN=r8_...           # clients read this env var automatically
+pip install 'replicate>=1,<2'               # pin: 2.0.0aN is alpha; unpinned can pull it
+npm install replicate                        # Node client
+```
+
+Python 3.8+ is required. Never `pip install replicate` unpinned in a Dockerfile or requirements
+file — a rebuild months later can resolve to the 2.0 alpha and break your imports. Why: the alpha is
+a full Stainless/httpx rewrite with a different surface.
+
+## Run a model
+
+```python
+import replicate
+
+output = replicate.run(
+    "black-forest-labs/flux-schnell",
+    input={"prompt": "a red bicycle", "num_outputs": 1},
+)
+```
+
+Since client 1.0.0, file outputs come back as **`FileOutput`** objects, not URL strings. Treating one
+as a string is the single most common bug.
+
+```python
+# Bad — output[0] is a FileOutput, not a str; this writes the repr, not the bytes
+open("out.png", "w").write(output[0])
+
+# Good — read the bytes, or take the URL explicitly
+with open("out.png", "wb") as f:
+    f.write(output[0].read())          # bytes
+print(output[0].url)                    # hosted URL if you'd rather link
+```
+
+Rule: call `.read()` for bytes or `.url` for the link. Why: silently coercing a `FileOutput` to a
+string corrupts the file and the error surfaces far from the cause.
+
+## Long-running & async
+
+For jobs over a few seconds, create a prediction instead of blocking:
+
+```python
+client = replicate.Client()
+prediction = client.predictions.create(
+    model="owner/model",
+    input={"prompt": "..."},
+)
+prediction.reload()                      # refresh status from the API
+while prediction.status not in ("succeeded", "failed", "canceled"):
+    time.sleep(2)
+    prediction.reload()
+```
+
+Set a **deadline** so a stuck job auto-cancels instead of billing forever, and `cancel()` on cleanup
+paths. Why: a hung prediction with no deadline is silent, open-ended GPU spend. Poll with a small
+backoff, not a tight loop — you are charged for the prediction, not the polling, but a tight loop
+wastes your own process and rate budget. Full polling loop with backoff and 5xx handling is in
+`references/webhooks-and-async.md`.
+
+## Webhooks
+
+For fire-and-forget, hand Replicate a URL and filter to the events you care about:
+
+```python
+client.predictions.create(
+    model="owner/model",
+    input={"prompt": "..."},
+    webhook="https://your.app/hooks/replicate",
+    webhook_events_filter=["completed"],   # not every intermediate "logs" event
+)
+```
+
+Rule: ALWAYS verify the signature before trusting a webhook body. Why: the URL is public — anyone can
+POST forged completions to it. Replicate signs each delivery; the secret has a **`whsec_`** prefix.
+Reconstruct signed content as `{webhook-id}.{webhook-timestamp}.{body}`, HMAC-SHA256 with the
+base64-decoded secret, base64-encode, and constant-time compare against the `webhook-signature`
+header (a space-separated `v1,<sig>` list). The clients expose a verification helper; the full
+Python and Node recipe (including idempotency via `webhook-id`) is in
+`references/webhooks-and-async.md`.
+
+## Deployments
+
+A **deployment** is a private, dedicated API endpoint for one model version that autoscales from zero
+to hundreds of instances. Reach for it when you need warm instances, a private endpoint, or a hard
+spend cap — not for one-off runs.
+
+- `min_instances` — the **warm floor**. Set >0 to kill cold starts for latency-sensitive traffic;
+  every warm instance bills whether or not it serves a request.
+- `max_instances` — the **spend cap**. The ceiling on concurrent instances; protects you from a
+  traffic spike turning into a surprise bill.
+- Hardware (NVIDIA T4, A100, H100, ...) is switchable **without touching code** — change the
+  deployment config, not `predict.py`.
+- Rolling updates ship a new version with no downtime; built-in monitoring covers latency,
+  throughput, error rate, and GPU memory.
+
+You still call a deployment with `run()` / `predictions.create()` — it just routes to your private
+instances. Create/update via HTTP API, the clients, or CLI; fields and the rolling-update flow are in
+`references/deployments-api.md`.
+
+## Package a custom model with Cog
+
+Cog packages a model into a production container. You need two files; Replicate builds the API server
+for you on push.
+
+```yaml
+# cog.yaml
+build:
+  gpu: true
+  python_version: "3.11"
+  python_packages:
+    - "torch==2.4.0"
+predict: "predict.py:Predictor"
+```
+
+```python
+# predict.py
+from cog import BasePredictor, Input, Path
+
+class Predictor(BasePredictor):
+    def setup(self):
+        # load weights ONCE here, not per request
+        self.model = load_model("weights.pth")
+
+    def predict(self, prompt: str = Input(description="text prompt")) -> Path:
+        result = self.model(prompt)
+        return Path(result)            # Cog uploads the file
+```
+
+```bash
+cog predict -i prompt="hello"          # run locally (needs Docker)
+cog push r8.im/owner/model             # build + push; Replicate hosts it
+```
+
+Rule: load weights in `setup()`, never in `predict()`. Why: `setup()` runs once per instance;
+`predict()` runs every request — loading weights per request makes every call pay the model-load cost.
+Full `cog.yaml` (system packages, run steps), typed `Input(...)`, GPU config, version pinning, and
+common build failures are in `references/cog-packaging.md`. Building requires Docker.
+
+## Cost & scaling levers
+
+- **Scale to zero** is the default — idle deployments stop billing. Only raise `min_instances` above
+  zero when cold-start latency actually hurts users, and treat the warm floor as a line item.
+- **Deadlines** on predictions cap the worst case so a stuck job cannot bill open-ended.
+- **Right-size the GPU**: do not run a 1B model on an H100. Hardware is a deployment-config change.
+- **Cache weights in `setup()`** so per-request work is just inference.
+
+This skill covers Replicate-specific levers only. Tracking total AI spend across many providers as a
+discipline is `cost-tracking`.
+
+## Anti-patterns
+
+| Anti-pattern | Why it bites | Do instead |
+|---|---|---|
+| Treating a `FileOutput` as a URL string | Corrupts files / writes a repr; error surfaces far away | `.read()` for bytes, `.url` for the link |
+| Blocking `run()` for a 10-min job in a web request | Request times out; the GPU job keeps running and billing | `predictions.create(webhook=...)`, return now |
+| Webhook handler with no signature check | The URL is public; anyone can forge completions | Verify HMAC-SHA256 against `whsec_` secret |
+| High `min_instances` "just in case" | Every warm instance bills 24/7 idle | Scale to zero; raise floor only when cold starts hurt |
+| Loading weights inside `predict()` | Every request pays the model-load cost | Load once in `setup()` |
+| `pip install replicate` unpinned | A rebuild can pull the 2.0 alpha and break imports | Pin `replicate>=1,<2` |
+| A deployment for a one-off run | Pays for a private endpoint you call once | Use the shared pool via `run()` |
+| No deadline on a long prediction | A hung job bills open-ended, silently | Set a deadline; `cancel()` on cleanup |
+
+## References
+
+- `references/cog-packaging.md` — full `cog.yaml` + `predict.py`, GPU config, build/push, version pinning, build failures.
+- `references/webhooks-and-async.md` — signature verification (Python + Node), event filters, idempotency, polling with backoff, 5xx retries.
+- `references/deployments-api.md` — create/update/get deployment via HTTP + clients, autoscaling fields, rolling updates, monitoring metrics, CLI.
+
+`scripts/verify.sh` statically checks an emitted artifact dir: a `cog.yaml` declaring `build:` and
+`predict:`, a `predict.py` `Predictor` with `setup` + `predict`, and warns on an unpinned `replicate`
+dependency or a `FileOutput` written as a string. Presence + key checks only — it does not run Docker.

package/skills/shared/replicate/evals/README.md

@@ -0,0 +1,9 @@
+# Evals — replicate
+
+`cases.yaml` is a trigger/routing and capability rubric, not an executable test of the Replicate API.
+`should_trigger` lists prompts (with a non-obvious symptom and a Spanish one) that must route here;
+`should_not_trigger` lists near-miss prompts that must route to a named sibling (`replicate-images`,
+`modal`, `runpod`, `webhooks`, `cost-tracking`) so the boundary stays sharp. The `capability` case is
+a scored rubric: feed the scenario to an agent loaded with this skill and check the response covers
+every `must_include` item. Run it through whatever eval harness scores the skills repo; no Replicate
+token, GPU, or network call is required to grade these.

package/skills/shared/replicate/evals/cases.yaml

@@ -0,0 +1,45 @@
+skill: replicate
+
+should_trigger:
+  - prompt: "Run flux-schnell from my Python script and save the image to disk"
+    why: "Core run + FileOutput handling — the most common entry point."
+  - prompt: "My Replicate prediction takes 8 minutes and my web request keeps timing out"
+    why: "Non-obvious symptom that routes to async predictions + webhook, not blocking run()."
+  - prompt: "Package my fine-tuned PyTorch model with Cog and push it to Replicate"
+    why: "Cog packaging — cog.yaml + predict.py + cog push."
+  - prompt: "How do I keep an instance warm so users don't hit a cold start on Replicate?"
+    why: "Non-obvious phrasing for deployments min_instances / warm floor."
+  - prompt: "Desplegar mi propio modelo en Replicate con un endpoint privado y autoescalado"
+    why: "Spanish phrasing for a deployment with a private endpoint and autoscaling."
+  - prompt: "How do I verify a webhook actually came from Replicate before trusting it?"
+    why: "Signature verification with the whsec_ secret — a security-critical ops task."
+  - prompt: "cog push fails with a CUDA/torch mismatch — how do I fix the build?"
+    why: "Cog build troubleshooting, covered in the cog-packaging reference."
+
+should_not_trigger:
+  - prompt: "What prompt and aspect ratio give the best product photo on flux?"
+    route_to: replicate-images
+    why: "Image prompt/param craft, not platform operations."
+  - prompt: "Run my own arbitrary Python on a serverless GPU, not a packaged Replicate model"
+    route_to: modal
+    why: "General serverless GPU containers, not the Replicate platform."
+  - prompt: "Rent a raw GPU box by the hour to train a model"
+    route_to: runpod
+    why: "Raw rented GPU infrastructure, not Replicate model serving."
+  - prompt: "Design a generic webhook receiver with retries and dedupe for Stripe events"
+    route_to: webhooks
+    why: "Provider-agnostic webhook consumer design, unrelated to Replicate signing."
+  - prompt: "Track our total AI spend across OpenAI, Anthropic, and Replicate as a discipline"
+    route_to: cost-tracking
+    why: "Cross-provider spend tracking; this skill covers only Replicate-specific levers."
+
+capability:
+  - scenario: "A team has a custom PyTorch model and wants it on Replicate as a private, autoscaling endpoint that fires webhooks on completion, while controlling GPU cost."
+    must_include:
+      - "cog.yaml declaring both build (gpu/python_version/python_packages) and predict"
+      - "predict.py with a Predictor whose setup() loads weights once and a typed predict()"
+      - "cog push r8.im/owner/model to publish a version"
+      - "a deployment configured with min_instances and max_instances and a hardware choice"
+      - "predictions.create with webhook= plus webhook_events_filter, and signature verification against the whsec_ secret"
+      - "cost guidance: scale-to-zero default vs warm-floor trade-off, prediction deadlines, right-sizing the GPU"
+      - "pins replicate>=1,<2 and treats file output as FileOutput (.read()/.url), not a string"

package/skills/shared/replicate/references/cog-packaging.md

@@ -0,0 +1,89 @@
+# Cog packaging — full reference
+
+Cog (open source, https://github.com/replicate/cog · https://cog.run) packages an ML model into a
+production Docker container. Two files drive it: `cog.yaml` (build + which predictor) and a Python
+module with a `BasePredictor`. `cog predict` runs it locally; `cog push r8.im/owner/model` builds and
+ships it, and Replicate auto-generates the API server on its GPU cluster. Building needs Docker.
+
+## Full `cog.yaml`
+
+```yaml
+build:
+  gpu: true                       # provisions a CUDA base image; omit/false for CPU-only
+  python_version: "3.11"          # quoted — "3.10" not 3.10
+  system_packages:                # apt packages baked into the image
+    - "ffmpeg"
+    - "libgl1-mesa-glx"
+  python_packages:                # pinned pip deps
+    - "torch==2.4.0"
+    - "transformers==4.44.0"
+    - "pillow==10.4.0"
+  run:                            # extra build steps, run after pip install
+    - "pip install --no-cache-dir flash-attn==2.6.3"
+predict: "predict.py:Predictor"   # module path : class name
+```
+
+Notes:
+- Quote `python_version` and every package version. Unpinned packages make builds non-reproducible.
+- `build.gpu: true` selects a CUDA image automatically — do not hand-install CUDA in `run:`.
+- `system_packages` is apt; `python_packages` is pip; `run:` is for anything neither covers.
+- For LLMs, you can add a `concurrency.max` to allow batched concurrent predict calls per instance.
+
+## Full `predict.py`
+
+```python
+from cog import BasePredictor, Input, Path
+import torch
+
+class Predictor(BasePredictor):
+    def setup(self) -> None:
+        """Runs ONCE per instance boot. Load weights here."""
+        self.device = "cuda" if torch.cuda.is_available() else "cpu"
+        self.model = torch.load("weights/model.pth", map_location=self.device)
+        self.model.eval()
+
+    def predict(
+        self,
+        prompt: str = Input(description="Text prompt"),
+        guidance: float = Input(description="Guidance scale", default=7.5, ge=1.0, le=20.0),
+        seed: int = Input(description="Random seed; -1 = random", default=-1),
+    ) -> Path:
+        """Runs PER request. Keep it pure inference."""
+        if seed == -1:
+            seed = int(torch.randint(0, 2**31 - 1, (1,)).item())
+        result = self.model.generate(prompt, guidance=guidance, seed=seed)
+        out = Path("/tmp/out.png")
+        result.save(out)
+        return out                  # Cog uploads it and returns a FileOutput to callers
+```
+
+Rules that matter:
+- **`setup()` once, `predict()` every call.** Weight loading, model compilation, and warmup belong in
+  `setup()`. Putting them in `predict()` makes every request pay the load cost.
+- Type every `predict()` argument and wrap it in `Input(...)` — Cog turns these into the model's API
+  schema, validation, and the website's input form. Use `default=`, `ge`/`le`, `choices=[...]`.
+- Return `Path` for file outputs (Cog uploads them), or a JSON-serializable value / `list[Path]`.
+- Bake weights into the image or download them in `setup()` to a cached path — never per request.
+
+## Local run, build, push
+
+```bash
+cog predict -i prompt="a cat" -i guidance=6.0     # run locally end to end (needs Docker)
+cog build -t my-model                              # build the image without pushing
+cog push r8.im/owner/model                         # build + push; creates a new version
+cog login                                          # if you haven't authenticated the CLI
+```
+
+Each `cog push` creates an immutable **version** (a SHA). Pin to a specific version in production calls
+(`owner/model:<version>`) so a later push does not silently change behavior; deployments target a
+version explicitly.
+
+## Common build failures
+
+| Symptom | Cause | Fix |
+|---|---|---|
+| `cog push` hangs / OOM on build | Huge weights copied into image | Download weights in `setup()` instead of baking, or use a slimmer base |
+| CUDA / torch version mismatch | `gpu: true` CUDA vs pinned torch wheel | Pin a torch build that matches the CUDA the image ships |
+| `predict.py:Predictor` not found | Wrong module path or class name in `predict:` | Match `file.py:ClassName` exactly |
+| Local works, push image broken | Relying on host files not in the image | Ensure weights/assets are in the build context or fetched in `setup()` |
+| `flash-attn` build error | Compiled against missing toolchain | Install via `run:` after torch, or use a prebuilt wheel |

package/skills/shared/replicate/references/deployments-api.md

@@ -0,0 +1,87 @@
+# Deployments API — full reference
+
+Source: https://replicate.com/docs/topics/deployments (accessed 2026-06-02).
+
+A deployment is a private, dedicated endpoint pointing at one model version. It autoscales from zero
+to hundreds of instances, keeps a warm floor if you ask, caps spend with a ceiling, and lets you swap
+hardware without touching `predict.py`. Updates roll out with no downtime.
+
+## Config fields
+
+| Field | Meaning | Cost/latency effect |
+|---|---|---|
+| `model` | `owner/model` the deployment serves | — |
+| `version` | the immutable version SHA to run | pin it; bump via update |
+| `hardware` | e.g. `gpu-t4`, `gpu-a100-large`, `gpu-h100` | bigger = faster + pricier |
+| `min_instances` | warm floor (>=0) | >0 kills cold starts but bills idle 24/7 |
+| `max_instances` | concurrency ceiling | the hard spend cap under spikes |
+
+Start `min_instances: 0` (scale to zero) and only raise it when measured cold-start latency hurts
+users. Set `max_instances` deliberately — it is your protection against a traffic spike becoming a
+surprise bill.
+
+## Create / update via HTTP API
+
+```bash
+# create
+curl -s -X POST https://api.replicate.com/v1/deployments \
+  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "name": "my-endpoint",
+    "model": "owner/model",
+    "version": "<version-sha>",
+    "hardware": "gpu-a100-large",
+    "min_instances": 1,
+    "max_instances": 5
+  }'
+
+# update (rolling — bump version or rescale with no downtime)
+curl -s -X PATCH https://api.replicate.com/v1/deployments/owner/my-endpoint \
+  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"version": "<new-version-sha>", "min_instances": 2}'
+```
+
+## Via the Python client
+
+```python
+client = replicate.Client()
+
+client.deployments.create(
+    name="my-endpoint",
+    model="owner/model",
+    version="<version-sha>",
+    hardware="gpu-a100-large",
+    min_instances=1,
+    max_instances=5,
+)
+
+# scale or roll a new version
+client.deployments.update("owner/my-endpoint", min_instances=2, version="<new-sha>")
+```
+
+## Calling a deployment
+
+You call it like any model — the deployment just routes to your private instances:
+
+```python
+deployment = client.deployments.get("owner/my-endpoint")
+prediction = deployment.predictions.create(input={"prompt": "..."})
+prediction.reload()
+```
+
+Async + webhook works the same: pass `webhook=` / `webhook_events_filter=` to the deployment's
+`predictions.create`.
+
+## Rolling updates
+
+Updating `version` rolls new instances in and drains old ones with no downtime. Because versions are
+immutable SHAs, you can roll back by updating `version` to the previous SHA. Never depend on "latest" —
+pin the SHA so a teammate's `cog push` does not silently change what the endpoint serves.
+
+## Monitoring
+
+The dashboard and API expose per-deployment **latency, throughput, error rate, and GPU memory**. Watch
+GPU memory to right-size hardware, and error rate after every rolling update — a spike right after a
+version bump means roll back to the prior SHA.

package/skills/shared/replicate/references/webhooks-and-async.md

@@ -0,0 +1,110 @@
+# Webhooks & async — full reference
+
+Source: https://replicate.com/docs/topics/webhooks/verify-webhook and the client READMEs
+(accessed 2026-06-02).
+
+## Event filters
+
+`webhook_events_filter` controls which lifecycle events POST to your URL. Filter to what you need —
+`["completed"]` for most apps, add `"start"`/`"output"`/`"logs"` only if you stream progress.
+
+```python
+client.predictions.create(
+    model="owner/model",
+    input={"prompt": "..."},
+    webhook="https://your.app/hooks/replicate",
+    webhook_events_filter=["completed"],
+)
+```
+
+## Signature verification — Python
+
+Replicate signs every delivery. The secret (fetch once from the API / dashboard) has a `whsec_`
+prefix. Signed content is `{webhook-id}.{webhook-timestamp}.{raw-body}`. Verify before trusting:
+
+```python
+import base64, hashlib, hmac
+
+def verify_replicate_webhook(headers: dict, raw_body: bytes, signing_secret: str) -> bool:
+    webhook_id = headers["webhook-id"]
+    timestamp  = headers["webhook-timestamp"]
+    signature_header = headers["webhook-signature"]   # e.g. "v1,<b64sig> v1,<b64sig2>"
+
+    secret_bytes = base64.b64decode(signing_secret.split("_", 1)[1])  # strip "whsec_"
+    signed_content = f"{webhook_id}.{timestamp}.{raw_body.decode()}".encode()
+    expected = base64.b64encode(
+        hmac.new(secret_bytes, signed_content, hashlib.sha256).digest()
+    ).decode()
+
+    # header may carry multiple space-separated "v1,<sig>" tokens; match any
+    for token in signature_header.split():
+        _, _, sig = token.partition(",")
+        if hmac.compare_digest(sig, expected):
+            return True
+    return False
+```
+
+- Use the **raw** request body bytes, not a re-serialized JSON dict — re-serialization changes bytes
+  and breaks the HMAC.
+- Always `hmac.compare_digest` (constant time), never `==` — `==` leaks timing.
+- Optionally validate the `webhook-timestamp` is recent (reject if older than ~5 min) to blunt replay.
+
+## Signature verification — Node
+
+```javascript
+import crypto from "node:crypto";
+
+export function verifyReplicateWebhook(headers, rawBody, signingSecret) {
+  const id = headers["webhook-id"];
+  const timestamp = headers["webhook-timestamp"];
+  const sigHeader = headers["webhook-signature"]; // "v1,<b64> v1,<b64>"
+
+  const secretBytes = Buffer.from(signingSecret.split("_")[1], "base64"); // strip "whsec_"
+  const signedContent = `${id}.${timestamp}.${rawBody}`;
+  const expected = crypto.createHmac("sha256", secretBytes)
+    .update(signedContent).digest("base64");
+
+  return sigHeader.split(" ").some((tok) => {
+    const sig = tok.split(",")[1];
+    const a = Buffer.from(sig);
+    const b = Buffer.from(expected);
+    return a.length === b.length && crypto.timingSafeEqual(a, b);
+  });
+}
+```
+
+The official clients also expose a built-in helper — prefer it when available; the manual recipe above
+is for frameworks where you only have raw headers + body.
+
+## Idempotency
+
+Webhooks can be delivered more than once. Dedupe on the `webhook-id` header (store processed ids,
+skip repeats) so a retry does not double-charge a user or re-run side effects.
+
+## 5xx and retries
+
+Return `2xx` fast once you have verified + enqueued the work. If your handler returns a `5xx`,
+Replicate retries — so keep the handler thin (verify, persist, enqueue) and do heavy work
+asynchronously. A handler that does slow work inline risks timing out and getting retried, compounding
+load.
+
+## Polling loop with backoff (when you don't use webhooks)
+
+```python
+import time
+
+def wait_for(client, prediction, max_wait=600):
+    deadline = time.monotonic() + max_wait
+    delay = 1.0
+    while prediction.status not in ("succeeded", "failed", "canceled"):
+        if time.monotonic() > deadline:
+            prediction.cancel()
+            raise TimeoutError("prediction exceeded max_wait")
+        time.sleep(delay)
+        delay = min(delay * 1.5, 10.0)   # capped exponential backoff
+        prediction.reload()
+    return prediction
+```
+
+Prefer webhooks for long jobs — polling ties up a process. Set a server-side prediction **deadline**
+too so a job auto-cancels even if your poller dies.