@dsalazardev/ai-hub-cli 0.0.0-stage → 0.1.0

package/skills/shared/machine-learning/references/metrics-and-imbalance.md

@@ -0,0 +1,82 @@
+# Metrics, thresholds, calibration, and imbalance (depth)
+
+Picking the metric and handling imbalance is where "0.99 accuracy" turns into an honest number. API current
+as of scikit-learn ~1.9 (verify at scikit-learn.org).
+
+## Choosing the classification metric
+
+Everything in `sklearn.metrics`. Score **probabilities** (`predict_proba`) for AUC/ranking metrics; score
+**labels** for precision/recall/F1 (which depend on the decision threshold).
+
+| You care about… | Metric | Notes |
+| --- | --- | --- |
+| Ranking, threshold-free, roughly balanced | `roc_auc_score` | Insensitive to the negative flood — flatters rare-positive problems. |
+| Ranking with **rare positives** (fraud/disease) | `average_precision_score` (PR-AUC) | The honest ranking metric under imbalance. |
+| Not crying wolf (false positives costly) | `precision_score` | TP / (TP+FP). |
+| Not missing cases (false negatives costly) | `recall_score` | TP / (TP+FN). |
+| Both, single number | `f1_score` | Harmonic mean; `average="macro"` treats classes equally, `"weighted"` by support. |
+| Per-class fairness, multiclass | `balanced_accuracy_score`, `f1_macro` | Macro-averaging stops the majority class dominating. |
+| The full picture | `confusion_matrix`, `classification_report` | Always look before trusting one scalar. |
+| Probability quality | `brier_score_loss`, `log_loss` | For calibrated-probability use cases. |
+
+Multiclass averaging: `micro` (global TP/FP counts — dominated by frequent classes), `macro` (unweighted
+class mean — each class equal), `weighted` (by class support). State which and why.
+
+## Regression metrics
+
+`r2_score` (proportion of variance explained; can be negative), `mean_absolute_error` (robust, same units),
+`root_mean_squared_error` (penalizes large errors; note this is the current function —
+`mean_squared_error(squared=False)` was removed), `mean_absolute_percentage_error` (scale-free but explodes
+near zero). Scoring strings negate losses: `"neg_root_mean_squared_error"`, `"neg_mean_absolute_error"`.
+
+## The threshold is a decision, not 0.5
+
+`predict()` thresholds probability at 0.5, which is rarely the business-optimal cut under imbalance or
+asymmetric costs. Tune it on validation, either manually from the PR curve or with the built-in estimator.
+
+```python
+from sklearn.metrics import precision_recall_curve
+prec, rec, thr = precision_recall_curve(y_val, proba_val)
+# e.g. smallest threshold that reaches precision >= 0.90:
+import numpy as np
+ok = np.where(prec[:-1] >= 0.90)[0]
+chosen = thr[ok[0]] if len(ok) else 1.0
+
+# Or let sklearn tune + wrap it (added 1.5):
+from sklearn.model_selection import TunedThresholdClassifierCV, FixedThresholdClassifier
+tuned = TunedThresholdClassifierCV(clf, scoring="f1", cv=5).fit(X_tr, y_tr)
+print(tuned.best_threshold_)
+fixed = FixedThresholdClassifier(clf, threshold=0.30).fit(X_tr, y_tr)  # pin a known-good cut
+```
+
+## Probability calibration
+
+Tree ensembles and SVMs produce distorted probabilities. If you use the probability itself (expected-value
+decisions, thresholds you trust), calibrate — on a held-out fold, never on train.
+
+```python
+from sklearn.calibration import CalibratedClassifierCV
+cal = CalibratedClassifierCV(clf, method="isotonic", cv=5)   # or method="sigmoid" (Platt)
+```
+
+## Imbalance — in order of preference
+
+1. **Do nothing to the data; fix the metric + threshold.** Often enough. Use PR-AUC and a tuned threshold.
+2. **`class_weight="balanced"`** (LogisticRegression, RandomForest, SVC, HistGradientBoosting…) or
+   `scale_pos_weight` (XGBoost) — reweight the loss, no synthetic rows, no leakage risk.
+3. **Resampling (SMOTE / undersampling) — only if 1–2 fall short, and only inside CV on the train fold.**
+   Resampling before the split, or touching the test fold, leaks and evaluates on synthetic data. Use the
+   **imbalanced-learn `Pipeline`** (not sklearn's) so the resampler runs per fold, train-only:
+
+```python
+from imblearn.pipeline import Pipeline as ImbPipeline
+from imblearn.over_sampling import SMOTE
+from sklearn.ensemble import HistGradientBoostingClassifier
+
+pipe = ImbPipeline([("pre", pre), ("smote", SMOTE(random_state=0)),
+                    ("clf", HistGradientBoostingClassifier(random_state=0))])
+# In cross_validate/GridSearchCV, SMOTE fits on each TRAIN fold only; test folds stay real.
+```
+
+Never resample the test set, and never report accuracy on the resampled (artificially balanced) data as if
+it were the real distribution.

package/skills/shared/machine-learning/references/pipelines-and-cv.md

@@ -0,0 +1,135 @@
+# Pipelines, custom transforms, and cross-validation (depth)
+
+Everything here keeps the SKILL.md rule: transforms that *learn* live inside the pipeline, so CV/tuning
+re-fit them on each train fold only. API current as of scikit-learn ~1.9 (verify at scikit-learn.org).
+
+## make_column_transformer — the terse ColumnTransformer
+
+`make_column_transformer` auto-names steps; `make_column_selector` picks columns by dtype/regex.
+
+```python
+from sklearn.compose import make_column_transformer, make_column_selector as mcs
+from sklearn.pipeline import make_pipeline
+from sklearn.impute import SimpleImputer
+from sklearn.preprocessing import StandardScaler, OneHotEncoder
+
+pre = make_column_transformer(
+    (make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), mcs(dtype_include="number")),
+    (make_pipeline(SimpleImputer(strategy="most_frequent"),
+                   OneHotEncoder(handle_unknown="ignore")), mcs(dtype_include=["object", "category"])),
+    remainder="drop",
+    verbose_feature_names_out=False,   # cleaner get_feature_names_out()
+)
+pre.set_output(transform="pandas")     # keep a DataFrame with real column names flowing through
+```
+
+## Stateless custom feature: FunctionTransformer
+
+For a transform with **no learned state** (log, ratio, date parts), `FunctionTransformer` is leak-safe by
+construction — there is nothing fit on data.
+
+```python
+import numpy as np
+from sklearn.preprocessing import FunctionTransformer
+
+log1p = FunctionTransformer(np.log1p, feature_names_out="one-to-one")
+```
+
+## Stateful custom transformer: fit on train only
+
+If the transform learns anything (a mean, a fitted mapping), write a class so CV re-fits it per fold.
+`fit` must store state from **train** only; `transform` applies it. Never compute the statistic in `__init__`
+or over the whole dataset.
+
+```python
+from sklearn.base import BaseEstimator, TransformerMixin
+
+class GroupRareCategories(BaseEstimator, TransformerMixin):
+    """Fold categories seen < min_count times in TRAIN into '__other__'."""
+    def __init__(self, min_count=10):
+        self.min_count = min_count
+    def fit(self, X, y=None):
+        counts = X.iloc[:, 0].value_counts()
+        self.keep_ = set(counts[counts >= self.min_count].index)   # learned on train fold
+        return self
+    def transform(self, X):
+        col = X.iloc[:, 0]
+        return col.where(col.isin(self.keep_), "__other__").to_frame()
+```
+
+## High-cardinality categoricals: TargetEncoder cross-fitting
+
+`sklearn.preprocessing.TargetEncoder` encodes each category by the target mean — the classic leakage trap
+if done naively. Its `fit_transform(X, y)` uses an **internal cross-fitting** scheme (each row encoded from
+*other* folds' targets), which is why `fit(X, y).transform(X)` deliberately differs and must not be used on
+train. Put it in the pipeline and let CV call `fit_transform`; never hand-roll a group-mean encode.
+
+```python
+from sklearn.preprocessing import TargetEncoder
+enc = TargetEncoder(smooth="auto", cv=5)   # cv controls the internal cross-fit
+# inside a ColumnTransformer branch for the high-cardinality columns
+```
+
+## Hyperparameter tuning — over the whole pipeline
+
+Address a step's parameter with `stepname__param` (double underscore). Tuning searches must wrap the
+*pipeline*, so preprocessing is re-fit each fold — tuning on globally-preprocessed data leaks.
+
+```python
+from sklearn.pipeline import Pipeline
+from sklearn.ensemble import HistGradientBoostingClassifier
+from sklearn.model_selection import RandomizedSearchCV, StratifiedKFold
+
+pipe = Pipeline([("pre", pre), ("clf", HistGradientBoostingClassifier(random_state=0))])
+param_dist = {
+    "clf__learning_rate": [0.01, 0.05, 0.1, 0.2],
+    "clf__max_leaf_nodes": [15, 31, 63],
+    "clf__l2_regularization": [0.0, 1.0, 10.0],
+}
+search = RandomizedSearchCV(
+    pipe, param_dist, n_iter=30,
+    cv=StratifiedKFold(5, shuffle=True, random_state=0),
+    scoring="average_precision", random_state=0, n_jobs=-1, refit=True,
+)
+search.fit(X_tr, y_tr)          # X_tr only — the test set stays sealed
+print(search.best_params_, search.best_score_)
+```
+
+- `GridSearchCV` — exhaustive; use for a small, discrete grid.
+- `RandomizedSearchCV` — sample `n_iter` combos from distributions; the default for anything nontrivial.
+- `HalvingRandomSearchCV` / `HalvingGridSearchCV` (`sklearn.model_selection`) — successive halving: cheap
+  runs on many candidates, more budget to survivors. Fastest when fits are expensive.
+
+## Nested CV — unbiased estimate of the tuning process
+
+If you report `search.best_score_` as your performance, it is optimistic (you selected on it). To estimate
+how the *whole tuning procedure* generalizes, nest an inner CV (tuning) inside an outer CV (scoring).
+
+```python
+from sklearn.model_selection import cross_val_score
+outer = StratifiedKFold(5, shuffle=True, random_state=1)
+nested = cross_val_score(search, X_tr, y_tr, cv=outer, scoring="average_precision")
+print(nested.mean(), nested.std())   # honest estimate of "tune-then-fit" on unseen data
+```
+
+## Splitters beyond StratifiedKFold
+
+```python
+from sklearn.model_selection import TimeSeriesSplit, GroupKFold, StratifiedGroupKFold
+
+# time order: train on past, test on future (expanding window); never shuffle the future into train
+for tr, te in TimeSeriesSplit(n_splits=5).split(X):
+    ...
+
+# clustered rows (same user/store): keep a group wholly in train OR test
+for tr, te in GroupKFold(n_splits=5).split(X, y, groups=user_id):
+    ...
+
+# clustered AND imbalanced: preserve class ratio while keeping groups intact
+for tr, te in StratifiedGroupKFold(n_splits=5).split(X, y, groups=user_id):
+    ...
+```
+
+Pass `groups=` through tuning too: `search.fit(X, y, groups=user_id)` with a group-aware `cv`. A random
+`KFold` on time-series or grouped data is leakage — CV will look great and production will not.
+```

package/skills/shared/modal/SKILL.md

@@ -0,0 +1,290 @@
+---
+name: modal
+description: "Use when running Python or GPU workloads serverlessly on Modal — modal.App, inline container Images, gpu= on @app.function, Volumes for weight caching, Cron schedules, ASGI endpoints, modal run vs serve vs deploy. NOT managed prediction APIs with no container of your own (that is replicate); NOT SSH-able GPU boxes rented by the hour (that is runpod)."
+tags: [modal, serverless, gpu, python, ai-infra, deployment, cron, web-endpoint]
+recommends: [replicate, runpod, fastapi, docker, python, llm-pipeline]
+origin: ai-hub
+---
+
+# Modal — serverless Python & GPU as decorators
+
+Modal runs your Python on remote containers without you ever writing a Dockerfile or a YAML
+file. The mental model: **infrastructure is declared inline as Python decorators.** A
+`modal.App` is the deployable unit; each `@app.function` runs in its own container built from
+a `modal.Image` you describe in code; you attach a GPU, a Volume, or a Secret as keyword
+arguments and the platform provisions, scales to zero, and tears down for you. There is no
+control plane to babysit — the source file *is* the infra.
+
+Pinned stack: **modal 1.4.3** (released 2026-05-18), Python **3.10–3.14** (`>=3.10,<3.15`).
+Install with `pip install modal` then `modal setup` to authenticate. Everything below uses
+the Modal 1.0+ API; several pre-1.0 forms were removed and are called out as Bad→Good.
+
+## Not this skill
+
+Modal owns the serverless-container-as-decorators surface and its CLI lifecycle; the *contents*
+of your function belong elsewhere.
+
+| The job | Goes to |
+|---|---|
+| Calling a **managed prediction API** with no container of your own | [`replicate`](../replicate/SKILL.md) / [`together-fireworks`](../together-fireworks/SKILL.md) / [`fal`](../fal/SKILL.md) |
+| Renting a **persistent, SSH-able GPU box** by the hour/week | [`runpod`](../runpod/SKILL.md) |
+| **FastAPI design** (routing, Pydantic, deps) independent of host | [`fastapi`](../fastapi/SKILL.md) |
+| Writing a **Dockerfile** for a registry / k8s / Compose | [`docker`](../docker/SKILL.md) |
+| General **Python language/runtime** questions | [`python`](../python/SKILL.md) |
+| **RAG / LLM pipeline** orchestration logic itself | [`llm-pipeline`](../llm-pipeline/SKILL.md) |
+
+## Decision: which entrypoint?
+
+| You want… | Use | Persists after exit? |
+|---|---|---|
+| Run a function once and exit (script, batch) | `modal run app.py` + `@app.local_entrypoint()` | No (ephemeral) |
+| Hot-reload dev loop for a web endpoint | `modal serve app.py` | No (dies on Ctrl-C) |
+| A persistent named deployment (prod, schedules, endpoints) | `modal deploy app.py` | Yes |
+| Fan out work across many containers | `.map()` / `.starmap()` / `.spawn()` inside an entrypoint | n/a |
+
+Rule: **schedules and live web endpoints require `modal deploy`.** `modal run` exits when the
+entrypoint returns, so a Cron defined under `modal run` never fires. `modal serve` is for the
+dev loop only — it watches your files and redeploys on save, but the app vanishes when you
+stop it.
+
+## The minimal app skeleton
+
+```python
+import modal
+
+app = modal.App("hello-modal")
+
+# The image is the container spec. Build it once, reuse across functions.
+image = modal.Image.debian_slim(python_version="3.12").uv_pip_install("requests")
+
+
+@app.function(image=image)
+def fetch(url: str) -> int:
+    import requests  # imported INSIDE the function: it lives in the remote image, not locally
+
+    return len(requests.get(url).content)
+
+
+@app.local_entrypoint()
+def main() -> None:
+    # Runs on your laptop; .remote() ships the call to a Modal container.
+    print(fetch.remote("https://modal.com"))
+```
+
+Run it: `modal run app.py`. **Bad** = wiring infra with argparse + a bash launcher + a
+hand-rolled Dockerfile. **Good** = the decorators above; the app, image, and scaling are all
+declared in the one file. Note the in-function import: dependencies you `uv_pip_install` exist
+in the *remote* image, so import them inside the function (or guard top-level imports), not at
+module top where your laptop would need them too.
+
+## Images — pin, layer, cache
+
+Build images by chaining methods on `modal.Image`. Rules, each with its why:
+
+1. **Prefer `.uv_pip_install(...)` over `.pip_install(...)`** — it resolves and installs with
+   `uv`, materially faster image builds.
+2. **Pin versions** — `.uv_pip_install("torch==2.5.1", "transformers==4.46.0")`. Unpinned
+   deps make builds non-reproducible and silently drift on rebuild.
+3. **Order layers stable→volatile** — system packages and big wheels first, your fast-changing
+   code last. Modal caches each layer; a change busts that layer and everything after it.
+4. **Add your own code with `.add_local_dir(...)` / `.add_local_python_source(...)`**, not by
+   pip-installing your repo. These are applied last so editing your source doesn't rebuild torch.
+5. **`.from_registry("...")`** when you need a specific base image; **`.apt_install("ffmpeg")`**
+   for system binaries; **`.run_commands(...)`** for arbitrary build steps.
+
+```python
+image = (
+    modal.Image.debian_slim(python_version="3.12")
+    .apt_install("ffmpeg")                                   # stable: rarely changes
+    .uv_pip_install("torch==2.5.1", "transformers==4.46.0")  # heavy wheels, pinned
+    .add_local_python_source("my_pkg")                       # volatile: your code, applied last
+)
+```
+
+→ [`references/images-gpu-cookbook.md`](references/images-gpu-cookbook.md) for vLLM / torch+CUDA
+/ diffusers recipes and the download-once weight-cache pattern.
+
+## GPU — it's a string now
+
+In Modal 1.0+ the GPU is a **string** on the decorator. The old `modal.gpu.H100()` objects
+were removed.
+
+- Single GPU: `gpu="H100"`.
+- Count via colon: `gpu="A100:2"` (two A100s in one container).
+- Memory variant: `gpu="A100-80GB"` (also `A100-40GB`).
+- Fallback list (first available wins): `gpu=["H100", "A100", "any"]`.
+- Supported types: `T4, L4, A10, L40S, A100(-40GB/-80GB), RTX-PRO-6000, H100, H200, B200`.
+
+```python
+# Bad — removed API, raises at import.
+# @app.function(gpu=modal.gpu.A100())
+
+# Good — string form.
+@app.function(image=image, gpu="A100-80GB", timeout=600)
+def embed(texts: list[str]) -> list[list[float]]: ...
+```
+
+Pick the smallest GPU that fits: **T4/L4** for cheap inference and small models, **A10/L40S**
+mid-range, **A100/H100** for training and large-model serving, **H200/B200** for frontier-scale.
+GPU time is billed per second a container is alive — never attach a GPU to a CPU-only job, and
+keep `scaledown_window` tight so idle GPU containers don't burn money.
+
+## Scaling & lifecycle
+
+Tune these keyword args on `@app.function`, each with its why:
+
+| Param | Effect | Why |
+|---|---|---|
+| `min_containers=N` | Keep N warm instances always running | Kills cold starts for latency-sensitive endpoints (costs idle compute) |
+| `buffer_containers=N` | Pre-warm N extra beyond current load | Smooths bursty traffic |
+| `scaledown_window=300` | Seconds an idle container lingers before shutdown | Reuse hot containers across nearby calls; lower = cheaper, higher = warmer |
+| `timeout=600` | Max seconds a single call may run | Caps runaway jobs |
+| `retries=3` | Auto-retry failed inputs | Survives transient failures in `.map()` fan-outs |
+
+Migration note: `keep_warm` → **`min_containers`** and `container_idle_timeout` →
+**`scaledown_window`** in the 1.0 migration. The old names are gone.
+
+Concurrency within a container is now its own decorator: **`@modal.concurrent(max_inputs=N)`**
+stacked under `@app.function` (it replaces the old `allow_concurrent_inputs=` argument). Use it
+so one container handles N simultaneous requests instead of one-per-container.
+
+## Volumes & Secrets
+
+A `Volume` is a distributed filesystem you mount into containers to persist data across runs —
+the canonical use is caching downloaded model weights so cold starts skip the re-download.
+
+```python
+weights = modal.Volume.from_name("hf-cache", create_if_missing=True)
+
+
+@app.function(image=image, gpu="H100", volumes={"/cache": weights})
+def serve_model():
+    # Reader: refresh the view so you see writes from other containers.
+    weights.reload()
+    # ... load model from /cache ...
+
+
+@app.function(image=image, volumes={"/cache": weights})
+def download_weights():
+    # ... write files into /cache ...
+    weights.commit()  # WITHOUT this, writes are NOT durable across containers
+```
+
+**Gotcha:** writers must call `vol.commit()` to persist; readers call `vol.reload()` to see
+another container's committed writes. Forgetting `commit()` is the #1 "my cache is empty"
+bug — the files existed in that container and vanished with it.
+
+Secrets land as **environment variables** in the container:
+
+```python
+@app.function(image=image, secrets=[modal.Secret.from_name("hf-token")])
+def pull():
+    import os
+
+    token = os.environ["HF_TOKEN"]  # value injected from the named Modal Secret
+```
+
+Never bake a token into the image (`.run_commands("export TOKEN=...")`) — it's recorded in
+layer history. Use a Secret. The cookbook above also carries the HF/OpenAI secret patterns.
+
+## Web endpoints
+
+Stack a web decorator **under** `@app.function`. Pick by surface:
+
+| Decorator | Use for | Needs |
+|---|---|---|
+| `@modal.fastapi_endpoint()` | A single GET/POST function-as-URL | `fastapi[standard]` in image |
+| `@modal.asgi_app()` | A full FastAPI/Starlette app you return | `fastapi[standard]` |
+| `@modal.wsgi_app()` | A Flask/Django WSGI app | the framework |
+| `@modal.web_server(port=8000)` | Your own server process (e.g. vLLM) on a port | the server |
+
+**Decorator stack order matters:** `@app.function` is outermost (top), then optional
+`@modal.concurrent`, then the web decorator innermost (bottom, closest to `def`).
+
+```python
+@app.function(image=image, gpu="H100", min_containers=1, scaledown_window=300)
+@modal.concurrent(max_inputs=10)   # middle
+@modal.asgi_app()                  # innermost
+def web():
+    from fastapi import FastAPI
+
+    api = FastAPI()
+
+    @api.get("/health")
+    def health():
+        return {"ok": True}
+
+    return api
+```
+
+Develop with `modal serve app.py` (hot-reload); ship with `modal deploy app.py` (stable URL).
+For custom domains, proxy-auth tokens, batching (`@modal.batched`), and concurrency tuning →
+[`references/web-and-scaling.md`](references/web-and-scaling.md). For the FastAPI app's *own*
+design (routes, Pydantic, deps), that's [`fastapi`](../fastapi/SKILL.md) — this skill only mounts it.
+
+## Scheduled jobs
+
+```python
+# Fixed wall-clock time, with timezone — survives redeploys at the same clock time.
+@app.function(schedule=modal.Cron("0 6 * * *", timezone="America/New_York"))
+def nightly_report(): ...
+
+
+# Interval relative to deploy time.
+@app.function(schedule=modal.Period(hours=5))
+def every_five_hours(): ...
+```
+
+**Gotcha:** `Period` is measured from deploy time and **resets on every redeploy** — redeploy
+at 4:59 and your "every 5 hours" clock restarts. `Cron` is wall-clock stable; prefer it for
+"run at 6am" semantics. Either way you must **`modal deploy`** (not `modal run`) for the
+schedule to live on the platform.
+
+## Parallelism
+
+Fan a function out across containers without managing a pool:
+
+```python
+@app.local_entrypoint()
+def main():
+    urls = ["https://a.com", "https://b.com", "https://c.com"]
+    # .map: one arg per call, results in input order.
+    sizes = list(fetch.map(urls))
+    # .starmap: each item is an argument tuple. .spawn: fire-and-forget -> handle.get() later.
+    handle = fetch.spawn("https://slow.com")
+    print(sizes, handle.get())
+```
+
+`.map(iterable)` returns results in order by default; pass `order_outputs=False` to yield as
+they complete (faster when latencies vary). Combine with `retries=` on the function so a
+single bad input doesn't sink the batch.
+
+## Anti-patterns
+
+| Anti-pattern | Do instead |
+|---|---|
+| "I'll use `gpu=modal.gpu.A100()` like the old docs" | Removed in 1.0. Use the string `gpu="A100-80GB"`. |
+| "Attach a GPU, it might speed up this CPU job" | GPU is billed per second alive. CPU-only job → no `gpu=`. |
+| "My files are written, the Volume will keep them" | Not without `vol.commit()` (writer) / `vol.reload()` (reader). |
+| "Pin later — `uv_pip_install('torch')` is fine for now" | Unpinned deps drift; builds aren't reproducible. Pin every version. |
+| "`modal run` it, the endpoint/schedule will stay up" | `run` is ephemeral; it exits. Use `modal deploy` for anything persistent. |
+| "Order the decorators however — Modal figures it out" | `@app.function` outermost, web decorator innermost. Wrong order errors. |
+| "Bake the HF token into the image with `run_commands`" | Leaks into layer history. Use `modal.Secret.from_name(...)`. |
+| "Just call the model via a managed API through Modal" | If you write no container, that's a managed-API job → [`replicate`](../replicate/SKILL.md). |
+| "I need a box to SSH into for a week" | That's a persistent rental → [`runpod`](../runpod/SKILL.md), not Modal's scale-to-zero. |
+| "Set `min_containers` high so it's always fast" | Idle warm containers cost money 24/7. Tune `scaledown_window` first. |
+
+## Verify
+
+[`scripts/verify.sh`](scripts/verify.sh) `[TARGET]` statically lints the nearest emitted Modal
+`*.py`: it requires a `modal.App(`, **fails** if the removed `modal.gpu.` object form appears,
+checks that any web decorator sits under an `@app.function`, and that any `Volume` uses
+`from_name(..., create_if_missing=...)`. It runs `python -c "import modal"` only if modal is
+installed (skip-pass otherwise), needs **no Modal credentials**, and exits 0 on an empty target.
+
+## Project grounding (02-DOCS)
+
+In a project with a `02-DOCS/` layer (the [`harness`](../harness/SKILL.md) wiki), read
+`02-DOCS/wiki/stack/modal.md` first, then record this app's real Modal choices there — GPU types,
+image base, Volume names, schedule, endpoint shape — and index it in `02-DOCS/wiki/index.md`. No
+`02-DOCS/`? Skip silently.

package/skills/shared/modal/evals/README.md

@@ -0,0 +1,29 @@
+# Eval harness — `modal` skill
+
+These cases are run by an **agent harness** (an LLM agent with the skill catalog available for
+routing), not a plain shell script. `cases.yaml` is the fixture; this file is the procedure.
+No Modal account or credentials are needed — every check is about *routing* and the *generated
+code rubric*, not live execution against Modal.
+
+## 1. Triggering accuracy
+
+Load only the skill descriptions (this `modal` one plus the bare names of routing siblings:
+replicate, runpod, fastapi, docker, python, llm-pipeline), not the bodies. For each prompt in
+`should_trigger` / `should_not_trigger`, ask which skill the agent would invoke, running 3–5
+trials per prompt. `should_trigger` passes if `modal` wins the majority; `should_not_trigger`
+passes if `modal` does **not** fire (bonus if it routes to the named `route_to` sibling). Pass
+bar: ≥90% of prompts, and no near-miss may fire `modal` in a majority of its trials. The
+Catalan/Spanish and the `gpu=modal.gpu.A100()` prompts are the ones to watch for flakiness.
+
+## 2. Capability uplift (with vs without)
+
+For each `capability` scenario, run the agent twice — once with no `modal` skill loaded, once
+with it fully loaded — and grade each output against the scenario's `must_include` checklist
+(a point counts only if genuinely correct, not merely mentioned). Pass bar: WITH-skill
+coverage ≥80% of the rubric, and WITH must clearly beat WITHOUT. The load-bearing points are
+the string GPU form, `Volume.from_name(create_if_missing=True)` with commit/reload, correct
+web-decorator stack order under `@app.function`, and `modal deploy` (not `run`) for anything
+persistent. Any rubric point the skill never produces is a gap to fix in `SKILL.md`.
+
+This is judgment-based, not byte-exact; record trial counts and the judge (human or model)
+alongside results for reproducibility.

package/skills/shared/modal/evals/cases.yaml

@@ -0,0 +1,50 @@
+skill: modal
+
+should_trigger:
+  - prompt: "Run my Python embedding job on an H100 without managing any servers — just ship the function and have it scale."
+    why: "Core serverless GPU on Modal: @app.function + gpu string + scale-to-zero. The defining use case."
+  - prompt: "My @app.function fails with gpu=modal.gpu.A100(). What's the right way to ask for that GPU now?"
+    why: "Non-obvious 1.0-migration error: the modal.gpu.X() object form was removed for the string form. Modal-specific symptom."
+  - prompt: "Cache my HuggingFace model weights across runs so cold starts don't re-download 16GB every time."
+    why: "Volume-backed weight-cache pattern (from_name + commit/reload) — squarely Modal state, phrased without naming Modal."
+  - prompt: "modal serve isn't hot-reloading my fastapi_endpoint changes — edits to the handler don't show up."
+    why: "Non-obvious dev-loop symptom tied to Modal's serve-vs-deploy lifecycle and web decorators."
+  - prompt: "Schedule a nightly scrape in the cloud at 6am New York time and keep it running."
+    why: "modal.Cron with timezone + the deploy-not-run gotcha for schedules to live. Cron/Period territory."
+  - prompt: "Desplegar un model de difusió amb GPU serverless i exposar-lo per HTTP amb endpoint propi."
+    why: "Catalan: serverless GPU + web endpoint (web_server/asgi_app) — the GPU-serving-over-HTTP core."
+  - prompt: "Ejecutar Python en GPU serverless que escale a cero y solo pague por segundo de uso."
+    why: "Spanish: scale-to-zero per-second GPU compute — Modal's billing/scaling model, not a managed API."
+
+should_not_trigger:
+  - prompt: "Run Stable Diffusion through a hosted prediction API where I just POST a prompt and get an image — no infra code."
+    route_to: "replicate"
+    why: "Managed prediction endpoint with no container of your own; Modal is for code-first containers."
+  - prompt: "Rent a persistent GPU pod I can SSH into and keep running for a week to iterate by hand."
+    route_to: "runpod"
+    why: "Persistent long-lived rentable GPU box; Modal is scale-to-zero serverless, not per-pod rental."
+  - prompt: "Design my FastAPI routers, Pydantic v2 models, and dependency injection for a user API."
+    route_to: "fastapi"
+    why: "FastAPI app design independent of host; Modal only mounts such an app, it doesn't design it."
+  - prompt: "Write a Dockerfile and push the image to a registry to deploy on Kubernetes."
+    route_to: "docker"
+    why: "Generic container/registry/k8s mechanics; Modal builds images inline in Python, no Dockerfile."
+  - prompt: "Structure my RAG retrieval, chunking, and reranking pipeline logic."
+    route_to: "llm-pipeline"
+    why: "LLM/RAG orchestration logic itself; Modal would only host it, the pipeline design is the sibling's."
+
+capability:
+  - scenario: "User asks: 'Build a Modal app that serves a vLLM endpoint on an H100 with a warm pool and a weight-cache volume, and tell me how to ship it.'"
+    must_include:
+      - "modal.App(...) defined as the deployable unit"
+      - "modal.Image with .uv_pip_install/.pip_install including vllm and fastapi[standard] (or equivalent server deps)"
+      - "GPU requested as the string form gpu=\"H100\" (NOT modal.gpu.H100())"
+      - "modal.Volume.from_name(..., create_if_missing=True) mounted via volumes={\"/path\": vol} for weight caching, with commit/reload noted"
+      - "Warm pool + lifecycle: min_containers and scaledown_window (and timeout) on @app.function"
+      - "A web decorator (@modal.web_server(port=...) for vLLM's own server, or @modal.asgi_app()) stacked UNDER @app.function in correct order"
+      - "Ship with `modal deploy` (not `modal run`) because the endpoint must persist"
+  - scenario: "User asks: 'Schedule a Python function to run every night at 6am New York time on Modal, and make sure it actually keeps firing.'"
+    must_include:
+      - "schedule=modal.Cron(\"0 6 * * *\", timezone=\"America/New_York\") on @app.function (Cron preferred over Period for wall-clock time)"
+      - "Note that Period resets on every redeploy whereas Cron is wall-clock stable (if Period is mentioned at all)"
+      - "Must `modal deploy` (not `modal run`) for the schedule to live on the platform"