@ericrisco/rsc 0.1.39 → 0.1.40

package/skills/nlp/references/evaluation.md

@@ -0,0 +1,107 @@
+# Evaluation — runnable metrics and when each one lies
+
+Depth overflow for SKILL.md section 3. All snippets use the HF `evaluate` library and
+`sacrebleu`; verify current API/package versions at author time. The governing rule: **pick the
+metric that matches the task, and never let an overlap metric be the final verdict on open-ended
+text.**
+
+## Classification — accuracy + macro-F1
+
+```python
+import evaluate
+acc = evaluate.load("accuracy")
+f1  = evaluate.load("f1")
+acc.compute(predictions=preds, references=labels)
+f1.compute(predictions=preds, references=labels, average="macro")   # macro on imbalance
+```
+
+- **accuracy** is honest only on balanced classes. With 95% negatives, "always predict negative"
+  scores 0.95 while being useless — **macro-F1** exposes that by averaging per-class F1 equally.
+- Report a confusion matrix too; a single scalar hides *which* class fails.
+
+## NER / token classification — entity-level F1 (seqeval)
+
+```python
+import evaluate
+seqeval = evaluate.load("seqeval")
+# predictions/references are lists of B-/I-/O tag STRINGS per token, specials already dropped
+seqeval.compute(predictions=[["B-PER","I-PER","O"]],
+                references =[["B-PER","I-PER","O"]])
+# -> precision / recall / f1 / accuracy at the ENTITY level, plus per-entity-type breakdown
+```
+
+- seqeval scores a **whole entity span** as correct only if every tag in it is right — the metric
+  that matches what NER is for. Per-token accuracy is inflated by the flood of `O` tokens and can
+  read ~0.97 on a model that misses half the entities.
+- Align labels to subwords first: first subword of a word gets the label; continuation subwords
+  and specials get `-100` (ignored by the loss); flip `B-XXX`→`I-XXX` on continuations.
+
+## Translation — BLEU / chrF (use sacrebleu for comparable numbers)
+
+```python
+import evaluate
+sacrebleu = evaluate.load("sacrebleu")
+sacrebleu.compute(predictions=["the cat sat on the mat"],
+                  references=[["the cat sat on the mat", "a cat was on the mat"]])
+# {'score': .., ...}  chrF: evaluate.load("chrf") — character n-gram F-score
+```
+
+- Use **sacrebleu**, not a hand-rolled BLEU — tokenization/normalization differences make raw
+  BLEU numbers non-comparable across papers; sacrebleu standardizes them.
+- **chrF** (character n-gram F-score) is often better than BLEU for **morphologically rich** and
+  agglutinative languages, because it credits partial-word overlap that word-BLEU misses.
+
+## Summarization — ROUGE
+
+```python
+import evaluate
+rouge = evaluate.load("rouge")
+rouge.compute(predictions=[summary], references=[reference])
+# rouge1 / rouge2 / rougeL — n-gram and longest-common-subsequence recall overlap
+```
+
+- ROUGE rewards overlap with the reference's n-grams, so it **favors extractive/copy-heavy**
+  summaries and is **blind to faithfulness** — a fluent summary that invents a fact can outscore a
+  faithful paraphrase. Pair it with a faithfulness/NLI check or an LLM judge.
+
+## The shared caveat: overlap metrics ≠ quality on open-ended text
+
+BLEU, ROUGE, and chrF correlate **weakly with human judgment** on open-ended and creative
+generation (documented across the MT and summarization evaluation literature). They reward
+matching the reference's *exact phrasing*: a correct paraphrase is punished, a wrong-but-fluent
+copy is rewarded. Legitimate use: **tracking regressions against a fixed reference set**, where a
+drop is a real signal. Illegitimate use: declaring model A "better" than B on open-ended output
+from a BLEU delta.
+
+## Perplexity (language modeling)
+
+```python
+import torch
+def perplexity(model, input_ids):
+    with torch.no_grad():
+        loss = model(input_ids, labels=input_ids).loss   # mean token NLL
+    return torch.exp(loss).item()
+```
+
+- Lower = the model predicts held-out text better. **Tokenizer-dependent**: PPL is defined over
+  the model's own tokens, so two models with different tokenizers/vocab have **non-comparable**
+  perplexities. Only compare within one tokenizer, on the same held-out set.
+
+## LLM-as-judge (the right tool for open-ended quality)
+
+```text
+Judge prompt (pin the rubric; score on fixed axes):
+  Rate the SUMMARY on 1–5 for: (a) faithfulness to SOURCE, (b) coverage, (c) fluency.
+  Output JSON: {"faithfulness": n, "coverage": n, "fluency": n, "reason": "..."}
+```
+
+- Known judge biases: **position** (favors the first candidate — randomize order), **verbosity**
+  (favors longer answers), **self-preference** (favors its own family's style). Mitigate: fixed
+  rubric, randomized order, and a **human spot-check** on a sample to calibrate the judge.
+- For agent/pipeline-level eval harnesses beyond single outputs, see `agent-eval`.
+
+## The one-change loop (applies to every metric above)
+
+Establish a baseline number on a **held-out set**, change exactly **one** variable (model OR
+tokenizer OR data OR decoding params), re-measure on the **same** set. Two changes at once and
+the delta is unattributable — and a regression can hide inside an improvement.

package/skills/nlp/references/tokenization.md

@@ -0,0 +1,95 @@
+# Tokenization — mechanics, special tokens, fertility
+
+Depth overflow for SKILL.md section 1. Grounded in the HF NLP course (chapters 6.4–6.8) and the
+`tokenizers` library; verify current API at author time (transformers current major ~v5).
+
+## The four subword algorithms
+
+All three "learned" algorithms share a preprocessing step (normalization + pre-tokenization);
+they differ in how they build and apply the vocabulary.
+
+### BPE (Byte-Pair Encoding) — GPT-2, RoBERTa, and many others
+
+1. Start from a base vocabulary (characters, or the 256 bytes for byte-level).
+2. Count all adjacent token pairs in the corpus; **merge the single most frequent pair** into a
+   new token; record the merge rule.
+3. Repeat until the vocab reaches the target size. The output is an ordered list of merge rules.
+
+To tokenize new text: split into base units, then **replay the merge rules in learned order**.
+
+**Byte-level BPE** (GPT-2, RoBERTa) runs BPE over raw **bytes** instead of Unicode characters:
+
+- Base vocabulary is exactly **256** (every byte value), so it stays tiny.
+- **No `[UNK]` token is ever needed** — every possible character maps to some byte sequence, so
+  emoji and unseen scripts are always representable (HF NLP course ch.6.5). This is the key
+  robustness win over character/word tokenizers.
+
+### WordPiece — BERT, DistilBERT, ELECTRA
+
+Similar merge-based training, but instead of picking the most *frequent* pair it picks the pair
+that maximizes a **likelihood score**: `score = freq(pair) / (freq(first) × freq(second))`. This
+favors merging pieces whose parts are individually rare — it prefers to build up rare tokens.
+
+To tokenize: **greedy longest-match from the front** of each word. Continuation subwords are
+prefixed with `##` (e.g. `token`, `##izers`). If no subword matches, the whole word becomes
+`[UNK]` — so WordPiece *can* lose OOV content, unlike byte-level BPE.
+
+### Unigram — T5, ALBERT, mT5, XLNet (usually via SentencePiece)
+
+Works **top-down**: start from a large candidate vocabulary and iteratively **remove** the tokens
+whose removal least increases the total corpus negative-log-likelihood, until the target size is
+reached. Each remaining token keeps a probability/score.
+
+To tokenize: choose the **segmentation with the highest product of token probabilities**
+(Viterbi over candidate splits), not a left-to-right greedy pass.
+
+### SentencePiece — the wrapper, not a fourth algorithm
+
+SentencePiece is a *framework* (usually running Unigram, sometimes BPE) that treats the input as
+a raw stream and encodes spaces as a normal meta-symbol `▁`. Consequences:
+
+- **Language-agnostic** — no whitespace pre-tokenizer, so it works on languages that don't
+  space-separate words (Chinese, Japanese, Thai).
+- **Reversible** — `decode` is literally "concatenate tokens and swap `▁`→space," so
+  `decode(encode(x)) == x` with no detokenization rules. This is why multilingual models favor it.
+
+## Special tokens (names differ by family — never hand-roll)
+
+| Purpose | BERT/WordPiece | RoBERTa/GPT byte-level | T5/SentencePiece |
+|---|---|---|---|
+| Start / pooled repr | `[CLS]` | `<s>` | (uses `pad` as decoder start) |
+| Separator / end | `[SEP]` | `</s>` | `</s>` |
+| Padding | `[PAD]` | `<pad>` | `<pad>` |
+| Unknown fallback | `[UNK]` | `<unk>` (rare, byte-level) | `<unk>` |
+| Mask (MLM only) | `[MASK]` | `<mask>` | — |
+
+- `[CLS]`/`<s>` position holds the sentence-level vector used by a classification head.
+- `[PAD]` tokens must be excluded from attention via the `attention_mask` the tokenizer returns;
+  forgetting this lets padding leak into the representation.
+- Add task-specific special tokens with `tokenizer.add_special_tokens(...)` **and** call
+  `model.resize_token_embeddings(len(tokenizer))`, or the new ids index past the embedding table.
+
+## Offset mapping (needed for NER / span tasks)
+
+`return_offsets_mapping=True` returns `(char_start, char_end)` per token so you can map predicted
+token spans back to characters in the original string — essential for entity extraction and for
+aligning B-/I- labels to subwords.
+
+```python
+enc = tok("Ada Lovelace", return_offsets_mapping=True)
+# offsets like [(0,0),(0,3),(4,12),(0,0)] — (0,0) marks added special tokens
+```
+
+## Measure fertility before committing (multilingual)
+
+Fertility = mean tokens per word. Compare it across your languages on a parallel sample; a large
+gap is the unfairness/cost tax from SKILL.md section 4.
+
+```python
+def fertility(texts, tokenizer):
+    toks = sum(len(tokenizer.tokenize(t)) for t in texts)
+    words = sum(len(t.split()) for t in texts)
+    return toks / words          # ~1.1–1.4 English on a good tokenizer; much higher off-target
+
+# Compare the SAME sentences across languages; a 2–15× gap = over-fragmentation of that script.
+```

package/skills/open-weights/SKILL.md

@@ -0,0 +1,239 @@
+---
+name: open-weights
+description: "Use when choosing an open-weight LLM and clearing it for use — which family and size fit a task, hardware and budget, and above all whether the license lets you ship (Llama, Qwen, Mistral/Mixtral, Gemma, DeepSeek, Phi, gpt-oss, and others). Owns the license-CLASS map (OSI-open Apache/MIT vs custom-community like Llama/Gemma vs non-commercial/restricted like the original Codestral) and the always-verify-the-model-card rule, plus size-to-VRAM budgeting and quant formats (GGUF/AWQ/GPTQ, what Q4_K_M means). Triggers: 'which open model should I use', 'can I use Llama/Gemma commercially', 'is Qwen Apache-2.0', 'what open LLM fits a 24GB GPU', 'is this license safe to ship in a product', 'quin model obert puc fer servir legalment'. NOT how to download or host on HF (that is huggingface), NOT running it locally (that is ollama), NOT serving at throughput (that is vllm), NOT fine-tuning it (that is finetuning) — this layer only tells you WHICH model and whether the license permits it, then hands off."
+tags: [open-weights, open-source-llm, model-selection, model-licenses, llama-qwen-mistral-gemma]
+recommends: [huggingface, ollama, finetuning, vllm]
+origin: risco
+---
+
+# Open weights — pick the model, then clear the license
+
+This is the CHOICE layer. Given a task, a box, and a shipping constraint, it tells you *which*
+open model to reach for and *whether the license lets you ship it*. It does not download, run,
+serve, or fine-tune anything — it feeds the skills that do (see [Related skills](#related-skills)).
+
+## The one rule (read before you name a model)
+
+**Model names and their licenses change monthly. Never state a license from memory — open the
+model card and confirm it, every time.** This file was authored against cards live in mid-2026;
+by the time you read it, versions have shipped and terms have moved. So:
+
+- Treat every specific version/license line below as **[verify the card]**, not as settled fact.
+- State the license **class** (OSI-open vs custom-community vs restricted), then send the reader to
+  the actual `LICENSE` / terms page for the exact model + size they intend to ship.
+- The *decision framework* and the *class taxonomy* are the durable parts. The version list is the
+  perishable part. Weight your trust accordingly.
+
+Real, verified-mid-2026 examples of *why* this rule exists — all confirmed against the source, and
+all things that were different at my Jan-2026 cutoff:
+
+- **Gemma 4** moved to **Apache-2.0**, while Gemma 1–3 stay on the custom Gemma Terms of Use
+  (confirmed on [ai.google.dev/gemma/terms](https://ai.google.dev/gemma/terms), accessed
+  2026-07-18). Same brand, opposite license class depending on version.
+- **Codestral 2** was relicensed **Apache-2.0** (per Mistral, Apr 2026), while the original
+  Codestral stays **non-production (MNPL)**. Same name, opposite shippability depending on release.
+- **DeepSeek-R1** is **MIT**, but the original **DeepSeek-V3** weights ship under a **custom
+  DeepSeek License Agreement with OpenRAIL-style use restrictions** (confirmed on the
+  [V3 LICENSE-MODEL](https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/LICENSE-MODEL)),
+  not MIT. Same family, different license per model.
+
+If three of the load-bearing facts moved in one release cycle, yours have too. Check the card.
+
+## 1. How to choose an open model
+
+Five gates, in this order. The license gate can veto everything above it, so never fall in love
+with a model before you clear it.
+
+1. **Task fit** — match the model's design to the job:
+   - General chat / instruct → Llama, Qwen, Mistral, Gemma instruct variants.
+   - Coding → Qwen-Coder, Codestral (license!), DeepSeek-Coder, gpt-oss.
+   - Reasoning / math → DeepSeek-R1-style reasoning models, gpt-oss, Qwen reasoning variants.
+   - Multilingual → Qwen and Gemma tend to lead; check the card's language list.
+   - Multimodal (vision) → Llama vision, Qwen-VL, Gemma vision, PaliGemma, Phi multimodal.
+   - Embeddings / retrieval → a dedicated embedding model, **not** a chat LLM → `embeddings-search`.
+   - Small / edge → 0.5B–4B (Qwen small, Gemma small, Phi-mini, gpt-oss-20b class).
+2. **Size vs your hardware class** — can it even fit? See [§4](#4-size--vram). A 70B at fp16 is
+   ~140 GB; if you have one 24 GB GPU, that model is off the table unless you quantize *and* accept
+   the quality/throughput cost. Pick a size your box can hold before you compare quality.
+3. **License / shippability** — **the gate that matters most** (see [§3](#3-licenses)). Decide up
+   front: is this a hobby/internal use, or a commercial product you distribute? "Open weights" does
+   **not** mean "open source" and does **not** guarantee commercial rights. Verify the card.
+4. **Hardware target** — consumer GPU (8–24 GB), data-center GPU (A100/H100 80 GB), Apple unified
+   memory, or CPU/edge. This narrows both size and quant format (GGUF for CPU/Mac/llama.cpp; AWQ/GPTQ
+   for GPU serving).
+5. **Community support** — a proxy for "will this actually work Monday morning": are there
+   ready-made GGUF/AWQ quants on the Hub, is it supported by your runtime (llama.cpp/Ollama/vLLM),
+   are there fine-tunes and recent downloads? A model with no quants and no runtime support is a
+   research artifact, not a shippable choice.
+
+Rule of thumb: **smallest model that passes your eval wins.** Don't reach for 70B when a well-chosen
+8B clears the bar — it's cheaper to run, faster, and fits more hardware. Prove it with `agent-eval`.
+
+## 2. Model families
+
+Families and *typical* size ladders below. Specific version claims are marked — **[verify]** at
+author time; the family and its rough sizing are the durable part. Full per-family notes,
+strengths, and license pointers in [references/model-families.md](references/model-families.md).
+
+| Family | Maker | Typical open sizes | Notes / current line [verify] |
+|---|---|---|---|
+| **Llama** | Meta | ~1B–~400B (dense + MoE) | Llama 4 (Scout/Maverick, MoE) current line; custom **Community License**, gated |
+| **Qwen** | Alibaba | 0.5B–235B+ (dense + MoE), strong Coder/VL | mostly Apache-2.0 but **per-size variance** — check each card |
+| **Mistral / Mixtral** | Mistral AI | 7B dense, 8x7B/8x22B MoE, "Small/Large" | open ones Apache-2.0; **Codestral original = MNPL non-production** |
+| **Gemma** | Google | ~1B–~27B, plus PaliGemma/CodeGemma/ShieldGemma | Gemma 1–3 = **custom Gemma Terms**; Gemma 4 = Apache-2.0 [verify] |
+| **DeepSeek** | DeepSeek | V3-class MoE (~600B+), R1 reasoning, distills | **per-model license split** — R1 MIT, original V3 custom; verify |
+| **Phi** | Microsoft | ~3B–15B ("mini", reasoning, multimodal) | **MIT** across the Phi-4 family [verify] |
+| **gpt-oss** | OpenAI | 20B and 120B (MoE, open *weights*) | **Apache-2.0** [verify]; OpenAI's first open-weight LLMs since GPT-2 |
+| Others | various | — | SmolLM, OLMo (fully-open incl. data), Falcon, Yi, Command — check card |
+
+## 3. Licenses
+
+**This is the load-bearing section. Get it wrong and you ship something you have no right to
+ship.** Every model falls into one of three classes. Identify the class, then open the card.
+
+### The three classes
+
+| Class | Examples (class, not a promise) | What it means for shipping |
+|---|---|---|
+| **OSI-open** (Apache-2.0, MIT) | Qwen (most), Mistral open, Phi, gpt-oss, DeepSeek-R1, Gemma **4** | Commercial use, modify, redistribute — permissive. Still read the card for attribution/notice. |
+| **Custom / community** | **Llama** (Meta Community), **Gemma 1–3** (Gemma Terms) | Broad free use **but with conditions**: acceptable-use policy, attribution, sometimes a scale cap. NOT OSI-open. |
+| **Non-commercial / restricted** | original **Codestral** (MNPL), any "research-only", some RAIL, some "≥$X revenue → buy a license" | You **cannot** ship it in a commercial product without a separate license. Fatal if missed. |
+
+"Open weights" describes availability of the weights file — it says nothing about your legal rights.
+A model can be a free download and still be non-commercial. **The download button is not a license.**
+
+### Per-family license notes (state the class, then verify the exact card)
+
+- **Llama = Meta Llama Community License — NOT OSI-open.** Carries an Acceptable Use Policy, a
+  **"Built with Llama"** attribution requirement on derivatives/products, naming rules, and a
+  **>700M-monthly-active-users clause** (above that you must request a separate license from Meta,
+  granted at Meta's discretion). Gated on the Hub (accept terms first). Verify the *current* Llama
+  version's terms — Meta has revised them across releases.
+- **Gemma = custom Gemma Terms of Use + Prohibited Use Policy** for Gemma 1–3 (not Apache), with a
+  duty to pass the restrictions **downstream** to every user and to ship the terms/notice file.
+  Gemma **4** reportedly moved to **Apache-2.0** — a textbook reason to check the version's card.
+- **Qwen = mostly Apache-2.0, but with per-size variance.** Historically some sizes (often the very
+  largest or a special tier) carried a separate Qwen license instead of Apache. **Check each size's
+  card** — do not assume "Qwen = Apache" for the whole family.
+- **Mistral / Mixtral open models = Apache-2.0**, but the **original Codestral is MNPL
+  (non-production)** — free for research/eval, **not** for commercial deployment. Codestral **2**
+  was relicensed Apache-2.0; verify which Codestral release you actually have. Note some newer
+  Mistral models use a "Modified MIT" with a revenue-threshold commercial clause — verify.
+- **DeepSeek = per-model split.** Code repos are MIT; **DeepSeek-R1 weights are MIT**; but the
+  original **DeepSeek-V3 weights** ship under a **custom DeepSeek License Agreement with OpenRAIL-
+  style use-based restrictions**. "DeepSeek = MIT" is too simple — verify the specific model.
+- **Phi = MIT** across the Phi-4 family [verify] — genuinely permissive, commercial-friendly.
+- **gpt-oss (OpenAI open-weight models, 20B/120B) = Apache-2.0** [verify] — permissive.
+
+### Commercial-use checklist (run before you ship)
+
+- [ ] Opened the **exact model + size** card and read its `LICENSE` / terms — not a blog, not this file.
+- [ ] Confirmed the **class**: OSI-open, custom-community, or restricted/non-commercial.
+- [ ] If custom/community: identified the **conditions** (attribution string, acceptable-use policy,
+      MAU/revenue caps, downstream pass-through duty) and can meet them.
+- [ ] If restricted: confirmed you are **not** deploying commercially, or obtained a separate license.
+- [ ] Checked whether the model is **gated** (accept terms on the Hub before download).
+- [ ] Checked the **base model** license — a fine-tune inherits the base's obligations (a Llama
+      fine-tune still owes "Built with Llama"; a distill can inherit the teacher's terms).
+
+Deeper class breakdown, gated-model mechanics, and the fine-tune-inheritance trap:
+[references/licenses.md](references/licenses.md).
+
+## 4. Size ↔ VRAM
+
+A **selection-time** budget: will this size class even fit your hardware? (Operational per-request
+KV-cache math lives in `ollama`/`vllm` — this is the "before I pick it" estimate.)
+
+```text
+weights_GB ≈ params(B) × bytes_per_param      # then add KV cache + runtime overhead on top
+  fp16/bf16 → 2.0   (≈ 2 GB per 1B params)
+  8-bit     → ~1.0  (≈ 1 GB per 1B params)
+  4-bit     → ~0.6  (≈ 0.6 GB per 1B + cache + overhead; conservative ~0.5–0.6/param)
+```
+
+What that means per hardware class (weights only — leave headroom for context/KV cache):
+
+| Hardware | fp16 ceiling | Practical pick (4-bit) |
+|---|---|---|
+| 8 GB consumer GPU | ~3B | 7–8B at 4-bit (~5 GB) |
+| 16 GB | ~7B | 13–14B at 4-bit |
+| 24 GB (e.g. 3090/4090) | ~10B | ~32B at 4-bit (~20 GB) |
+| 80 GB (A100/H100) | ~34B | 70B at 4-bit (~40 GB), or a big-MoE like gpt-oss-120b |
+| Apple unified (e.g. 64 GB) | shares with the OS | budget against **total** unified memory |
+| multi-GPU / 2×80 GB+ | 70B fp16 (~140 GB) via tensor-parallel | full-precision large models |
+
+**A 70B at fp16 is ~140 GB — it does not fit one GPU.** You either quantize (4-bit ≈ ~40 GB, fits one
+80 GB card) or split across GPUs with **tensor parallelism** (that's a `vllm` job). Longer context
+grows the KV cache *on top of* weights, so keep a margin. Full derivation + KV math:
+[references/sizing-and-quant.md](references/sizing-and-quant.md).
+
+## 5. Quant / formats
+
+Quantization shrinks weights (fewer bits/param) to fit smaller hardware, trading a little quality.
+Which **format** you pick is driven by your runtime:
+
+| Format | Where it runs | Use it for |
+|---|---|---|
+| **GGUF** | llama.cpp, **Ollama**, LM Studio | CPU, Apple Silicon, single-box local; the everyday local format |
+| **AWQ** | vLLM, TGI, transformers | GPU serving — activation-aware 4-bit, strong quality/latency |
+| **GPTQ** | vLLM, TGI, transformers | GPU serving — older, widely available 4-bit |
+| bitsandbytes (NF4) | transformers | quick load-time 4/8-bit for experiments/fine-tuning |
+| EXL2 | ExLlamaV2 | flexible bit-rates on consumer GPUs |
+
+**Decoding a GGUF quant tag like `Q4_K_M`:** `Q4` = ~4-bit weights; `_K` = a k-quant (mixed
+precision — keeps the more sensitive tensors at higher bit-depth); `_M` = the medium size/quality
+tier (`_S` smaller/lower, `_L` larger/higher). **`Q4_K_M` is the standard default** — roughly half
+the memory of fp16 for a few percent quality loss. `Q8_0` is near-lossless (~1 byte/param); below
+`Q4`, quality drops off fast. Don't go sub-Q4 to force a too-big model onto a too-small box — pick a
+smaller model instead. More in [references/sizing-and-quant.md](references/sizing-and-quant.md).
+
+## 6. Where to get it + how to run it (routing)
+
+You've chosen a model and cleared its license. Now hand off — this skill stops here.
+
+- **Get the weights / host on HF** → `huggingface` (Hub download/upload, Inference Providers,
+  endpoints). Also where you convert to GGUF.
+- **Run it locally on one box** (laptop/desktop, GGUF via Ollama) → `ollama`.
+- **Serve it at throughput / with tensor-parallel** (production, batching, big models) → `vllm`.
+- **Fine-tune it** (LoRA/QLoRA, SFT, preference tuning) → `finetuning`.
+- **Prove the small model is enough** before committing → `agent-eval`.
+
+The Hub filters (task + license + size + recent downloads) and the Ollama library are where you
+actually find candidates — that mechanics lives in `huggingface` / `ollama`, not here.
+
+## Guardrails / gotchas
+
+- **Never assert a license from memory.** State the class, open the card. Licenses change per
+  version, per size, and per release month.
+- **"Open weights" ≠ "open source" ≠ "commercial-use OK".** Three different things.
+- **A fine-tune inherits its base license.** Fine-tuning Llama does not launder away "Built with
+  Llama" or the acceptable-use policy; a distill can inherit the teacher model's terms.
+- **Gated ≠ restricted.** Gemma/Llama are gated (accept terms to download) yet usable commercially
+  under their conditions; a "research-only" card is restricted regardless of gating.
+- **Don't sub-Q4 a too-big model onto a too-small GPU.** Quality collapses and it still may OOM at
+  real context. Choose a size that fits.
+- **Newest ≠ best for you.** The smallest model that passes your eval wins on cost, speed, and fit.
+- **MoE total vs active params.** A "120B" MoE may activate only ~5B/token (fast) but you still need
+  VRAM for *all* experts — size against total params, not active.
+
+## Related skills
+
+- **`huggingface`** — get/host: Hub download/upload, Inference Providers, endpoints, GGUF
+  conversion. This skill tells you *which* repo to pull; `huggingface` pulls it.
+- **`ollama`** — run one model locally (GGUF, single box, VRAM sizing at request time). This skill
+  says which model + quant *class* fits; `ollama` runs it and does the KV-cache math.
+- **`vllm`** — serve at throughput, batching, tensor-parallel for models too big for one GPU.
+- **`finetuning`** — adapt a chosen base with LoRA/QLoRA/SFT. Choose+clear the base here first.
+- Boundary: this skill is **selection + licensing knowledge only**. It never re-teaches calling,
+  running, serving, or training — it feeds those four.
+
+## Checklist
+
+- [ ] Named the **task** and matched a family designed for it (chat/code/reasoning/vision/embeddings/edge).
+- [ ] Picked the **smallest size** likely to pass the eval, and confirmed it **fits the hardware class**.
+- [ ] Identified the **license class** (OSI-open / custom-community / restricted).
+- [ ] Opened the **exact model + size card** and confirmed the license — not from memory, not from a blog.
+- [ ] For commercial use: met the conditions (attribution, acceptable-use, MAU/revenue caps) or confirmed permissive.
+- [ ] Checked the **base model** license if it's a fine-tune/distill.
+- [ ] Picked a **quant format** matching the runtime (GGUF local / AWQ-GPTQ serving) and a sane tier (`Q4_K_M` default).
+- [ ] Handed off: `huggingface` (get), `ollama` (run local), `vllm` (serve), `finetuning` (adapt).

package/skills/open-weights/evals/README.md

@@ -0,0 +1,14 @@
+# Evals for the `open-weights` skill
+
+Routing + capability checks read from `cases.yaml` by the skill eval harness — no network, no
+model downloads, no license fetches. The `should_trigger` prompts must select this skill: they
+cover general model selection, the Llama commercial-license question, Qwen per-size license
+variance, VRAM-fit sizing, the non-commercial (Codestral/MNPL) gate, and a Catalan phrasing.
+The `should_not_trigger` prompts must route to the named real sibling instead — `huggingface`
+(download/host), `ollama` (run local + KV cache), `vllm` (serve/tensor-parallel), `finetuning`
+(adapt), and `rag` (retrieval) — proving the tight "choose only, then hand off" boundary. The
+single `capability` case is a rubric: a good answer to the 24GB commercial-assistant scenario must
+verify the license on the exact card (never from memory), classify the license class, flag Llama as
+custom/non-OSI with its conditions, size the model to fit 24GB, note fine-tune license inheritance,
+and hand off to the four sibling skills. Run with whatever harness loads `evals/cases.yaml`;
+nothing to install or connect.

package/skills/open-weights/evals/cases.yaml

@@ -0,0 +1,42 @@
+skill: open-weights
+
+should_trigger:
+  - prompt: "Which open-weight model should I pick for a customer-support chatbot I plan to ship commercially?"
+    why: "Core selection + license-clearance question across families — the CHOICE layer this skill owns."
+  - prompt: "Can I use Llama commercially in my SaaS product, or does its license restrict that?"
+    why: "License-class question on Llama (Meta Community License, not OSI-open) — the load-bearing section."
+  - prompt: "Is Qwen Apache-2.0 across every size, or do I have to check each model card separately?"
+    why: "Per-size license variance — squarely the verify-the-card rule this skill teaches."
+  - prompt: "What is the largest open LLM I can realistically run on a single 24GB GPU?"
+    why: "Size-to-VRAM budgeting to narrow the model choice before picking — this skill's §4."
+  - prompt: "Is the original Codestral safe to use in a commercial product, or is it non-commercial only?"
+    why: "Restricted/non-commercial (MNPL) gate — the fatal-if-missed license class."
+  - prompt: "Quin model obert puc fer servir legalment en un producte comercial i que càpiga en una GPU de 16GB?"
+    why: "Catalan (non-English) phrasing; open-model selection + license + VRAM fit — this skill's whole remit."
+
+should_not_trigger:
+  - prompt: "Download the Llama weights from the Hub with the hf CLI and push a model card."
+    route_to: huggingface
+    why: "Getting/hosting weights on HF — huggingface's job; this skill only says WHICH repo, not how to pull it."
+  - prompt: "My 70B OOMs in Ollama at 128K context — how do I shrink the KV cache to make it fit?"
+    route_to: ollama
+    why: "Operational, single-box run + per-request KV-cache math — ollama, not the selection layer."
+  - prompt: "Serve a 70B across two 80GB GPUs with tensor parallelism and high request throughput."
+    route_to: vllm
+    why: "Production serving engine + tensor-parallel flags — vllm owns serving, not model choice."
+  - prompt: "Fine-tune Qwen on my instruction dataset with LoRA and export the adapter."
+    route_to: finetuning
+    why: "Adapting a chosen base with LoRA/SFT — finetuning; this skill only chooses+clears the base first."
+  - prompt: "Build a retrieval pipeline over our docs with chunking, an index, and reranking."
+    route_to: rag
+    why: "Designing a retrieval system, not selecting or licensing an open model."
+
+capability:
+  - scenario: "Choose an open-weight model for a commercial customer-support assistant that must run on a single 24GB GPU, and confirm its license lets you ship."
+    must_include:
+      - "verify the license on the EXACT model + size card, never assert it from memory — licenses change per version, size, and month"
+      - "classify the license: OSI-open (Apache-2.0/MIT) vs custom-community (Llama Meta Community, Gemma 1-3) vs non-commercial/restricted"
+      - "Llama is NOT OSI-open: Meta Community License with an acceptable-use policy, a 'Built with Llama' attribution requirement, and a >700M-monthly-active-users clause"
+      - "pick a size that fits 24GB: ~32B at 4-bit (~20GB) or an 8-14B, leaving headroom for KV cache — a 70B at fp16 (~140GB) does not fit one GPU"
+      - "a fine-tune or distill inherits its base model's license obligations"
+      - "hand off after choosing: huggingface to download/host, ollama to run locally, vllm to serve, finetuning to adapt"

package/skills/open-weights/references/licenses.md

@@ -0,0 +1,123 @@
+# Open-weight LICENSES — the load-bearing reference
+
+**Get this wrong and you ship something you have no legal right to ship.** This is the section that
+most often bites teams. The cardinal rule, again: **state the license class, then open the exact
+model + size card and confirm.** Everything below is class guidance + [verify-the-card] pointers,
+authored mid-2026 — licenses move per version, per size, per release month.
+
+## The three classes
+
+### 1. OSI-open (Apache-2.0, MIT) — permissive
+
+Commercial use, modification, and redistribution are allowed. Apache-2.0 adds a patent grant and a
+notice/attribution requirement (keep the `NOTICE`/`LICENSE`); MIT is barely-there attribution. This
+is the class you want for a product you distribute. Still read the card — attribution and notice
+obligations are real even when permissive.
+
+Class members (mid-2026, **verify each**): most Qwen sizes, Mistral open models, Phi (MIT), gpt-oss
+(Apache-2.0), DeepSeek-R1 (MIT), Gemma **4** (Apache-2.0), OLMo, SmolLM.
+
+### 2. Custom / community — free use *with conditions*
+
+Broad, royalty-free rights **but** the vendor keeps strings attached: an acceptable-use policy,
+mandatory attribution, naming rules, sometimes a scale cap, and a duty to pass restrictions
+downstream. **This is NOT OSI-open** even though people loosely call it "open." You can usually ship
+commercially — *if* you meet every condition.
+
+Class members (mid-2026, **verify**): **Llama** (Meta Community License), **Gemma 1–3** (Gemma
+Terms of Use).
+
+### 3. Non-commercial / restricted — cannot ship without a separate license
+
+Free to download and evaluate, but **not** for commercial deployment. Includes "research-only"
+cards, some RAIL variants, revenue-gated licenses ("above $X/month you must buy a commercial
+license"), and non-production licenses. Missing one of these is the expensive mistake.
+
+Class members (mid-2026, **verify**): **original Codestral** (MNPL non-production), some Cohere
+Command terms, various research-only community fine-tunes.
+
+## Per-family detail
+
+### Llama — Meta Llama Community License (custom, NOT OSI-open)
+
+- **Acceptable Use Policy** incorporated by reference — prohibited uses bind you and your users.
+- **Attribution:** derivatives/products must **prominently display "Built with Llama"**, and derived
+  model names must carry "Llama" per the naming rules.
+- **Redistribution:** include a copy of the Community License with any distribution of the weights.
+- **Scale cap:** if your product (or affiliates') exceeds **700M monthly active users** in the prior
+  calendar month, you must **request a license from Meta**, granted at Meta's sole discretion.
+- **Gated** on the Hub — accept terms before you can download.
+- Meta has revised terms across Llama versions — **verify the current version's license page**.
+
+### Gemma — custom Gemma Terms of Use + Prohibited Use Policy (Gemma 1–3)
+
+- Not Apache. A Google-authored license with a **Prohibited Use Policy** covering sensitive domains.
+- **Downstream pass-through:** if you distribute Gemma or a Gemma-derived model (weights, API, or
+  embedded), you must include/reference the Terms, ship the notice file ("Gemma is provided under
+  and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms"), and bind downstream
+  users to the Prohibited Use Policy. The restrictions **follow derivatives** — a fine-tune can't be
+  used for a purpose that would've been prohibited on the base.
+- **Gemma 4** reportedly switched to **Apache-2.0** (confirmed on ai.google.dev/gemma/terms,
+  2026-07-18). Same brand, different class by version — **check which Gemma you have.**
+- Google claims no rights in outputs you generate.
+
+### Qwen — mostly Apache-2.0, with per-size variance
+
+- Most sizes are Apache-2.0. Historically the very largest (and occasionally a special tier) shipped
+  under a separate **Qwen license**. Do not assume the whole family is Apache — **check each size**.
+
+### Mistral / Mixtral
+
+- Open models (Mistral 7B, Mixtral 8x7B/8x22B, many "Small") = **Apache-2.0**.
+- **Original Codestral = MNPL (Mistral AI Non-Production License)** — research/eval only, **no
+  commercial deployment**. **Codestral 2** relicensed **Apache-2.0** (Apr 2026) [verify which one].
+- Some newer models use a **"Modified MIT"** with a revenue-threshold commercial clause (e.g. above
+  a monthly-revenue bar you need a commercial license / Mistral platform access). Verify.
+
+### DeepSeek — per-model split
+
+- **Code** repositories: MIT.
+- **DeepSeek-R1** weights: **MIT**.
+- **Original DeepSeek-V3** weights: **custom DeepSeek License Agreement** (v1.0) with **OpenRAIL-
+  style use-based restrictions** (military use, harm to minors, disinformation, PII, discrimination,
+  etc.), which you must **pass forward** as an enforceable provision on redistribution.
+- Newer point releases have trended toward MIT [verify]. "DeepSeek = MIT" is an over-simplification.
+
+### Phi — MIT
+
+- The Phi-4 family (Phi-4, Phi-4-mini, reasoning, multimodal) is **MIT** [verify] — commercial use,
+  fine-tuning, redistribution allowed, no royalties.
+
+### gpt-oss — Apache-2.0
+
+- gpt-oss-20b / gpt-oss-120b = **Apache-2.0** [verify] — permissive commercial use.
+
+## Gated models (mechanics)
+
+Some models (Llama, Gemma, many others) are **gated**: you must accept terms on the Hub model page,
+then use a read-scoped token. Gating is an access gate, not a use-class — a gated model can be
+commercially usable (under its conditions), and an ungated model can still be non-commercial.
+Downloading without accepting terms returns a 403. The download step itself is a `huggingface` job.
+
+## The fine-tune / distill inheritance trap
+
+**A derivative inherits the base model's license and obligations.**
+
+- Fine-tune Llama → your fine-tune still owes "Built with Llama", the Acceptable-Use Policy, and the
+  MAU clause. You cannot re-license it as MIT.
+- Fine-tune Gemma → the Gemma Terms + Prohibited Use Policy follow your model downstream.
+- A **distill** trained on another model's outputs can inherit the **teacher's** terms (this is how
+  a restriction can travel into a model whose own card looks permissive). Check the base_model chain.
+- Merging models can compound obligations from **every** parent. Trace the lineage.
+
+When in doubt, treat the strictest ancestor's license as governing, and confirm on each card.
+
+## Commercial-ship decision flow
+
+1. Identify the **exact model + size** (not the family) you'll deploy.
+2. Open its `LICENSE` / terms page on the Hub or the vendor site.
+3. Classify: OSI-open → ship (honor attribution/notice). Custom/community → meet the conditions
+   (attribution string, AUP, MAU/revenue caps, pass-through) → ship. Restricted/non-commercial →
+   **stop**; get a separate license or pick another model.
+4. If it's a fine-tune/distill/merge, repeat for **every** ancestor and take the strictest.
+5. Record the decision (which model, which license, which conditions) — see `decision-records`.