@dsalazardev/ai-hub-cli 0.0.0-stage → 0.1.0

package/skills/shared/nlp/SKILL.md

@@ -0,0 +1,225 @@
+---
+name: nlp
+description: "Use when choosing how to tokenize text or which transformer type fits an NLP task, when a tokenizer over-fragments non-English text or inflates token cost, when picking a language metric, or when classification, NER or summarization output looks wrong and it is unclear whether the tokenizer, the architecture or the metric is at fault. Covers subword tokenizers, encoder versus decoder versus encoder-decoder choice, sentence embeddings, and the metric families. NOT retrieval or vector search (that is `embeddings-search`), NOT the RAG loop (that is `rag`), NOT prompt wording (that is `prompt-engineering`), NOT training the network (that is `finetuning`)."
+tags: [nlp, tokenization, bpe, wordpiece, ner, text-classification, bleu, rouge, perplexity, transformers]
+recommends: [deep-learning, embeddings-search, rag, training-data]
+origin: ai-hub
+---
+
+# nlp — tokenize the text, pick the model type, pick the metric
+
+You own the **language-modeling discipline**: how raw text becomes tokens, which transformer
+architecture fits a task, and which metric actually tells you whether it worked. When the
+question is "which tokenizer," "BERT or GPT or T5 for this," "why does my Catalan text cost 3×
+the tokens," or "is this BLEU score meaningful," this is the skill. You stop at retrieval, the
+RAG loop, prompt wording, and the training step itself — those route out (below).
+
+## Route out first (loud — do not duplicate these)
+
+- **Retrieval embeddings + vector search** (which embedding model, chunking, recall@k, rerank)
+  → [`../embeddings-search/SKILL.md`](../embeddings-search/SKILL.md). Sentence embeddings *live
+  here* as a task; using them to *retrieve* is theirs.
+- **The retrieve → prompt → generate → answer loop** and groundedness → [`../rag/SKILL.md`](../rag/SKILL.md).
+- **Prompt wording / few-shot / system prompts** → [`../prompt-engineering/SKILL.md`](../prompt-engineering/SKILL.md).
+- **Training the network** (LoRA/SFT, trainer loop, PyTorch) → [`../finetuning/SKILL.md`](../finetuning/SKILL.md)
+  + [`../deep-learning/SKILL.md`](../deep-learning/SKILL.md).
+- **The training corpus itself** (JSONL messages, label sets) → [`../training-data/SKILL.md`](../training-data/SKILL.md).
+
+## Decision: model type per task (get this right before anything else)
+
+Pick the architecture from the task's *shape*, not from what is trendy. A decoder LLM can
+technically classify, but a fine-tuned encoder is smaller, faster, cheaper, and usually more
+accurate on a fixed-label task.
+
+| Task shape | Architecture | Why | Example families* |
+|---|---|---|---|
+| Understand / label a whole input (classification, NER, extractive QA, similarity) | **Encoder** (bidirectional) | Attends to the full sentence both directions; cheap to fine-tune and to serve | BERT, RoBERTa, DistilBERT, ModernBERT |
+| Free-form generation, chat, few-shot | **Decoder** (autoregressive) | Attends only to prior tokens; predicts the next token | GPT-style, Llama, Gemma, Qwen |
+| Transform input → new text (summarize, translate, generative QA) | **Encoder-decoder / seq2seq** | Encoder reads all of the source, decoder writes conditioned on it | T5 / FLAN-T5, BART, mT5 |
+
+\* Architecture families are stable; specific checkpoints and their licenses are not — check the
+HF model card before you commit (licenses change; see `open-weights`). ModernBERT (2024) is a
+current long-context encoder; verify the latest at author time.
+
+The two most common own-goals: reaching for a 7B decoder to do sentiment on 5 classes (an
+encoder does it for a fraction of the cost), and forcing an encoder to *generate* (it cannot —
+it has no decoder).
+
+## 1. Tokenization
+
+Every downstream number depends on this step, and its failures are **silent**. The single
+load-bearing rule:
+
+> **Load the tokenizer that shipped with the checkpoint, and use the same one at train and
+> inference.** `AutoTokenizer.from_pretrained(same_checkpoint)`. A train/inference tokenizer
+> mismatch — different vocab, different special tokens, different casing/normalization — maps
+> text to token ids the model never saw and corrupts everything downstream with no error.
+
+```python
+from transformers import AutoTokenizer   # transformers current major ~v5 (verify at author time)
+
+tok = AutoTokenizer.from_pretrained("bert-base-cased")
+enc = tok("Tokenizers matter.", return_offsets_mapping=True)
+tok.convert_ids_to_tokens(enc["input_ids"])
+# ['[CLS]', 'Token', '##izers', 'matter', '.', '[SEP]']  — note WordPiece '##' continuation + added specials
+```
+
+**The four algorithms** (full mechanics in `references/tokenization.md`):
+
+| Algorithm | Builds vocab by… | Applies by… | Used by |
+|---|---|---|---|
+| **BPE** | merging the most frequent adjacent pair, repeatedly | split to chars, replay learned merges | GPT-2 (byte-level), many |
+| **WordPiece** | merging pairs that maximize a likelihood score | longest-match subword from the front (`##` continuations) | BERT family |
+| **Unigram** (SentencePiece) | start large, *remove* tokens that least hurt corpus likelihood | most-probable segmentation | T5, ALBERT, mT5 |
+| **Byte-level BPE** | BPE over the 256 raw **bytes**, not Unicode chars | same as BPE on bytes | GPT-2, RoBERTa |
+
+- **Byte-level BPE has no `[UNK]`.** Base vocab is exactly 256 (all byte values), so every
+  emoji, accent, and script maps to *some* byte sequence — nothing falls out as unknown
+  (verified: HF NLP course ch.6). WordPiece/word-level tokenizers *do* have `[UNK]` and lose OOV
+  content.
+- **SentencePiece is reversible** — it treats space as a normal symbol (the `▁` meta-symbol), so
+  `decode(encode(x)) == x` without language-specific detokenization rules. That is why it
+  dominates multilingual models.
+
+**Special tokens are not decoration.** `[CLS]`/`<s>` carries the pooled sentence
+representation for classification; `[SEP]`/`</s>` marks segment/end; `[PAD]` fills a batch (and
+must be masked out via `attention_mask`); `[MASK]` is the MLM target; `[UNK]` is the fallback.
+Names differ by model (`[CLS]` in BERT vs `<s>` in RoBERTa) — another reason to never hand-roll
+the tokenizer.
+
+**Why it matters — three concrete costs:**
+
+- **$ cost & context.** Token count *is* the bill and the context budget. Fewer tokens per
+  sentence = cheaper calls and more room in the window.
+- **OOV / information loss.** A tokenizer that emits `[UNK]` throws away content it can't
+  represent; byte-level/SentencePiece degrade gracefully instead.
+- **Fairness.** Vocab trained mostly on English fragments other scripts far harder — the same
+  meaning costs more tokens, more money, and more latency (section 5).
+
+## 2. Tasks
+
+### Text classification (encoder + classification head)
+```python
+from transformers import pipeline
+clf = pipeline("text-classification", model="distilbert-base-uncased-finetuned-sst-2-english")
+clf("The service was slow but the food was incredible.")
+# [{'label': 'POSITIVE', 'score': 0.99...}]
+```
+Metric: accuracy on balanced data; **macro-F1** the moment classes are imbalanced (accuracy
+lies when 95% of rows are one class).
+
+### Token classification / NER (encoder, per-token labels)
+Labels are **B-/I-/O** spans aligned to subword tokens: the first subword of a word gets the
+label, continuation subwords and special tokens get `-100` (ignored by the loss). Evaluate with
+**seqeval** at the **entity** level, never per-token accuracy (per-token accuracy is inflated by
+the flood of `O` tokens).
+```python
+from transformers import pipeline
+ner = pipeline("token-classification", aggregation_strategy="simple")
+ner("Ada Lovelace worked in London.")
+# groups subwords back into entities: PER 'Ada Lovelace', LOC 'London'
+```
+
+### Seq2seq — summarization / translation (encoder-decoder)
+```python
+summ = pipeline("summarization", model="facebook/bart-large-cnn")
+summ(long_article, max_length=130, min_length=30)
+```
+Metric: **ROUGE** for summarization, **BLEU/chrF** for translation — with the heavy caveat in
+section 4.
+
+### Sentence embeddings (SBERT — the *task*, not retrieval)
+```python
+from sentence_transformers import SentenceTransformer   # sentence-transformers ~v5 (verify)
+model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
+emb = model.encode(["The weather is lovely today.", "It's so sunny outside!"])
+model.similarity(emb, emb)   # semantic textual similarity / clustering / paraphrase mining
+```
+Producing/judging embeddings **for retrieval** (model choice, chunking, recall@k, rerankers) is
+[`embeddings-search`](../embeddings-search/SKILL.md), not here.
+
+## 3. Evaluation — pick the metric that matches the task
+
+| Task | Primary metric | Catches | Trap |
+|---|---|---|---|
+| Classification | accuracy + **macro-F1** | wrong labels | accuracy hides minority-class failure |
+| NER / token | **entity-level F1** (seqeval) | missed/partial spans | per-token accuracy is inflated by `O` |
+| Translation | **BLEU / chrF** | n-gram overlap w/ reference | weak on meaning; chrF better for morphology |
+| Summarization | **ROUGE** (1/2/L) | recall of reference n-grams | rewards copying; blind to faithfulness |
+| Generation (LM) | **perplexity** | how well the model predicts held-out text | tokenizer-dependent — not comparable across tokenizers |
+| Open-ended / chat | **LLM-as-judge** + human | quality overlap metrics miss | judge bias (position, verbosity, self-preference) |
+
+**The caveat that governs this whole section:** BLEU, ROUGE, and chrF are n-gram/character
+overlap metrics and **correlate weakly with human judgment on open-ended and creative
+generation** — they reward matching the reference's exact phrasing, so a correct paraphrase
+scores low and a fluent-but-wrong copy scores high (well documented; e.g. the summarization and
+MT-evaluation literature). Use them for **regression tracking on a fixed reference set**, never
+as the final verdict on quality. For open-ended output, use an **LLM-as-judge rubric plus a
+human spot-check** — and know the judge has its own biases (position, verbosity, self-preference),
+so pin the rubric and randomize order.
+
+**Perplexity** = exp(mean token NLL): lower means the model predicts held-out text better. It is
+**tokenizer-dependent**, so two models with different tokenizers have non-comparable perplexities
+— only compare within the same tokenizer/vocab. Runnable snippets for seqeval, sacrebleu, ROUGE,
+perplexity, and an LLM-judge harness are in `references/evaluation.md`.
+
+## 4. Multilingual pitfalls
+
+The English-centric trap: a tokenizer whose vocab was learned mostly on English **over-fragments
+other scripts**. The same sentence in Ukrainian, Arabic, Hindi, or even accented Catalan can take
+**2–15× more tokens** than its English equivalent (Petrov et al., *Language Model Tokenizers
+Introduce Unfairness Between Languages*, NeurIPS 2023). That "fertility" (tokens per word)
+inflation is a triple tax:
+
+- **Money** — more tokens per identical meaning = a proportionally larger bill for the same work.
+- **Context** — over-fragmented text eats the window faster, so fewer few-shot examples fit and
+  long documents truncate sooner.
+- **Quality** — sequences fragmented into byte-shards are harder to model, degrading accuracy for
+  exactly the users the tool already serves worst.
+
+Mitigations: prefer a **multilingual tokenizer/model** (mT5, XLM-R, a SentencePiece-based model)
+whose vocab actually covers your languages; **measure fertility** on your own corpus (tokens per
+word, per language) before you commit; and don't benchmark cost or latency only on English.
+
+## Guardrails / gotchas
+
+- **Tokenizer must match the checkpoint, at train and inference.** Mismatch corrupts silently,
+  no error. The most expensive bug in this skill.
+- **Encoders can't generate; don't classify with a giant decoder by default.** Match architecture
+  to task shape.
+- **BLEU/ROUGE/chrF ≠ quality on open-ended text.** Overlap metrics; weak human correlation. Track
+  regressions with them; judge quality with an LLM-judge + human.
+- **Entity-F1 (seqeval), not token accuracy, for NER.** `O` tokens inflate accuracy toward 1.0.
+- **macro-F1, not accuracy, on imbalanced classes.**
+- **Perplexity is tokenizer-relative** — never compare it across different tokenizers.
+- **Don't benchmark tokenization/cost only in English** — fertility varies 2–15× across scripts.
+- **Never assert a model's license from memory** — check the current model card; license classes
+  shift (Llama = Meta Community license, not OSI-open; Gemma = custom terms; etc.).
+
+## Related skills
+
+- [`embeddings-search`](../embeddings-search/SKILL.md) — retrieval embeddings, chunking, recall@k,
+  reranking. NLP owns *making/judging* sentence embeddings as a task; using them to search is theirs.
+- [`rag`](../rag/SKILL.md) — the full retrieve→generate answer loop and groundedness.
+- [`prompt-engineering`](../prompt-engineering/SKILL.md) — the wording of the prompt.
+- [`finetuning`](../finetuning/SKILL.md) + [`deep-learning`](../deep-learning/SKILL.md) — actually
+  training/adapting the network (this skill picks the type and metric; those move the weights).
+- [`training-data`](../training-data/SKILL.md) — building the labeled corpus you train on.
+
+## Checklist
+
+- [ ] Architecture picked from task shape (encoder / decoder / enc-dec), not habit.
+- [ ] Tokenizer loaded from the *same* checkpoint, used identically at train and inference.
+- [ ] Tokenizer choice justified vs OOV, cost, and — if multilingual — measured fertility.
+- [ ] Special tokens and `attention_mask` handled (padding masked, `-100` on ignored labels).
+- [ ] Metric matches the task: macro-F1 (imbalanced), entity-F1/seqeval (NER), ROUGE/BLEU/chrF
+      only as a regression signal, LLM-judge + human for open-ended.
+- [ ] Perplexity compared only within one tokenizer.
+- [ ] Retrieval / RAG / prompt / training concerns routed to the sibling skill, not re-solved here.
+
+## References
+
+- `references/tokenization.md` — BPE/WordPiece/Unigram/byte-level training mechanics, special
+  tokens per family, offset mapping, and a fertility-measuring snippet.
+- `references/evaluation.md` — runnable seqeval, sacrebleu (BLEU/chrF), ROUGE, perplexity, and an
+  LLM-as-judge rubric, with when each lies.

package/skills/shared/nlp/evals/README.md

@@ -0,0 +1,10 @@
+# Evals — nlp
+
+Routing and capability checks for a human or LLM grader, not an automated suite. Read each
+`should_trigger` prompt and confirm an agent would reach for `nlp` (tokenizer choice, model-type
+decision, task modeling, or metric selection); read each `should_not_trigger` prompt and confirm
+it routes to the named sibling (`embeddings-search`, `rag`, `prompt-engineering`, `finetuning`,
+`training-data`) for the stated reason. Grade the `capability` scenario by checking the produced
+answer against every `must_include` bullet — the current API/method, the load-bearing guardrail
+(tokenizer mismatch, fertility tax), and the honesty hedge (BLEU/ROUGE weak on open-ended gen,
+licenses must be checked). A pass covers all routing cases plus full rubric coverage.

package/skills/shared/nlp/evals/cases.yaml

@@ -0,0 +1,45 @@
+skill: nlp
+
+should_trigger:
+  - prompt: "Should I use BPE or WordPiece for my tokenizer, and what's the difference?"
+    why: "Core tokenization-algorithm choice this skill owns; no retrieval or training context."
+  - prompt: "BERT vs GPT vs T5 — which one do I pick for classifying support tickets into 6 categories?"
+    why: "Model-type-per-task decision (encoder vs decoder vs enc-dec) is the skill's opening decision gate."
+  - prompt: "My tokenizer splits Catalan text into way more tokens than the English equivalent, so it costs more — why?"
+    why: "Multilingual over-fragmentation / fertility tax is section 4; the cost symptom points squarely here."
+  - prompt: "Is BLEU or ROUGE a meaningful metric for judging my summaries, or is it misleading?"
+    why: "Metric selection plus the weak-correlation caveat for open-ended generation, owned in section 3."
+  - prompt: "How do I evaluate a NER model properly — entity-level F1 vs per-token accuracy?"
+    why: "Token-classification evaluation with seqeval entity-F1 is explicitly the skill's task + metric."
+  - prompt: "quin model faig servir per classificar text: un encoder tipus BERT o un LLM generatiu?"
+    why: "Catalan trigger on the encoder-vs-decoder model-type decision, the skill's primary gate."
+  - prompt: "My fine-tuned classifier gives garbage at inference but was fine in training — could the tokenizer be the problem?"
+    why: "Non-obvious: the classic train/inference tokenizer mismatch that corrupts silently, the skill's load-bearing guardrail."
+
+should_not_trigger:
+  - prompt: "Which embedding model and chunk size should I use to make semantic search return relevant results?"
+    route_to: embeddings-search
+    why: "Retrieval embedding choice, chunking and recall@k are the retrieval technique layer, not task modeling."
+  - prompt: "Build the full RAG pipeline: retrieve context, prompt the model, and generate a grounded answer."
+    route_to: rag
+    why: "End-to-end retrieve→generate answer loop and groundedness live in rag, past where nlp stops."
+  - prompt: "Rewrite my system prompt with few-shot examples so the model stops refusing valid requests."
+    route_to: prompt-engineering
+    why: "Prompt wording / few-shot design, not tokenization, architecture, or metrics."
+  - prompt: "Fine-tune Llama with LoRA on my instruction dataset using trl and peft."
+    route_to: finetuning
+    why: "Actually training/adapting the network is the finetuning method skill, not picking type/metric."
+  - prompt: "Convert my customer chats into a JSONL messages dataset with system/user/assistant turns for training."
+    route_to: training-data
+    why: "Building the training corpus format is training-data, not NLP task modeling or metrics."
+
+capability:
+  - scenario: "A team has 40k multilingual support tickets (English + Catalan + Arabic) to auto-route into 8 categories and also to extract product names as entities. They're unsure whether to use a big generative LLM, why Arabic tickets seem to cost far more tokens, and how to know if the models are any good."
+    must_include:
+      - "Picks an encoder (BERT-family / ModernBERT, multilingual e.g. XLM-R) with a classification head for the 8-way routing rather than a large generative decoder, justifying cost/latency/accuracy, and hedges that specific checkpoints and licenses must be checked on the model card"
+      - "Loads the tokenizer from the same checkpoint and uses it identically at train and inference, calling out that a train/inference tokenizer mismatch corrupts results silently"
+      - "Explains the Arabic token blow-up as tokenizer over-fragmentation / fertility (2–15x more tokens on non-English scripts), recommends a multilingual SentencePiece-based tokenizer and measuring fertility per language before committing"
+      - "Uses a token-classification (NER) setup with B-/I-/O labels aligned to subwords (-100 on continuations/specials) for product-name extraction"
+      - "Evaluates routing with macro-F1 (not raw accuracy, because classes are imbalanced) and NER with entity-level F1 via seqeval (not per-token accuracy)"
+      - "Notes that if any free-text summarization/quality judgment is added, BLEU/ROUGE/chrF correlate weakly with human judgment on open-ended output and an LLM-as-judge plus human spot-check is needed, with judge-bias mitigation"
+      - "Routes retrieval/vector search to embeddings-search, the RAG answer loop to rag, prompt wording to prompt-engineering, and the actual training to finetuning/deep-learning"

package/skills/shared/nlp/references/evaluation.md

@@ -0,0 +1,107 @@
+# Evaluation — runnable metrics and when each one lies
+
+Depth overflow for SKILL.md section 3. All snippets use the HF `evaluate` library and
+`sacrebleu`; verify current API/package versions at author time. The governing rule: **pick the
+metric that matches the task, and never let an overlap metric be the final verdict on open-ended
+text.**
+
+## Classification — accuracy + macro-F1
+
+```python
+import evaluate
+acc = evaluate.load("accuracy")
+f1  = evaluate.load("f1")
+acc.compute(predictions=preds, references=labels)
+f1.compute(predictions=preds, references=labels, average="macro")   # macro on imbalance
+```
+
+- **accuracy** is honest only on balanced classes. With 95% negatives, "always predict negative"
+  scores 0.95 while being useless — **macro-F1** exposes that by averaging per-class F1 equally.
+- Report a confusion matrix too; a single scalar hides *which* class fails.
+
+## NER / token classification — entity-level F1 (seqeval)
+
+```python
+import evaluate
+seqeval = evaluate.load("seqeval")
+# predictions/references are lists of B-/I-/O tag STRINGS per token, specials already dropped
+seqeval.compute(predictions=[["B-PER","I-PER","O"]],
+                references =[["B-PER","I-PER","O"]])
+# -> precision / recall / f1 / accuracy at the ENTITY level, plus per-entity-type breakdown
+```
+
+- seqeval scores a **whole entity span** as correct only if every tag in it is right — the metric
+  that matches what NER is for. Per-token accuracy is inflated by the flood of `O` tokens and can
+  read ~0.97 on a model that misses half the entities.
+- Align labels to subwords first: first subword of a word gets the label; continuation subwords
+  and specials get `-100` (ignored by the loss); flip `B-XXX`→`I-XXX` on continuations.
+
+## Translation — BLEU / chrF (use sacrebleu for comparable numbers)
+
+```python
+import evaluate
+sacrebleu = evaluate.load("sacrebleu")
+sacrebleu.compute(predictions=["the cat sat on the mat"],
+                  references=[["the cat sat on the mat", "a cat was on the mat"]])
+# {'score': .., ...}  chrF: evaluate.load("chrf") — character n-gram F-score
+```
+
+- Use **sacrebleu**, not a hand-rolled BLEU — tokenization/normalization differences make raw
+  BLEU numbers non-comparable across papers; sacrebleu standardizes them.
+- **chrF** (character n-gram F-score) is often better than BLEU for **morphologically rich** and
+  agglutinative languages, because it credits partial-word overlap that word-BLEU misses.
+
+## Summarization — ROUGE
+
+```python
+import evaluate
+rouge = evaluate.load("rouge")
+rouge.compute(predictions=[summary], references=[reference])
+# rouge1 / rouge2 / rougeL — n-gram and longest-common-subsequence recall overlap
+```
+
+- ROUGE rewards overlap with the reference's n-grams, so it **favors extractive/copy-heavy**
+  summaries and is **blind to faithfulness** — a fluent summary that invents a fact can outscore a
+  faithful paraphrase. Pair it with a faithfulness/NLI check or an LLM judge.
+
+## The shared caveat: overlap metrics ≠ quality on open-ended text
+
+BLEU, ROUGE, and chrF correlate **weakly with human judgment** on open-ended and creative
+generation (documented across the MT and summarization evaluation literature). They reward
+matching the reference's *exact phrasing*: a correct paraphrase is punished, a wrong-but-fluent
+copy is rewarded. Legitimate use: **tracking regressions against a fixed reference set**, where a
+drop is a real signal. Illegitimate use: declaring model A "better" than B on open-ended output
+from a BLEU delta.
+
+## Perplexity (language modeling)
+
+```python
+import torch
+def perplexity(model, input_ids):
+    with torch.no_grad():
+        loss = model(input_ids, labels=input_ids).loss   # mean token NLL
+    return torch.exp(loss).item()
+```
+
+- Lower = the model predicts held-out text better. **Tokenizer-dependent**: PPL is defined over
+  the model's own tokens, so two models with different tokenizers/vocab have **non-comparable**
+  perplexities. Only compare within one tokenizer, on the same held-out set.
+
+## LLM-as-judge (the right tool for open-ended quality)
+
+```text
+Judge prompt (pin the rubric; score on fixed axes):
+  Rate the SUMMARY on 1–5 for: (a) faithfulness to SOURCE, (b) coverage, (c) fluency.
+  Output JSON: {"faithfulness": n, "coverage": n, "fluency": n, "reason": "..."}
+```
+
+- Known judge biases: **position** (favors the first candidate — randomize order), **verbosity**
+  (favors longer answers), **self-preference** (favors its own family's style). Mitigate: fixed
+  rubric, randomized order, and a **human spot-check** on a sample to calibrate the judge.
+- For agent/pipeline-level eval harnesses beyond single outputs, see `agent-eval`.
+
+## The one-change loop (applies to every metric above)
+
+Establish a baseline number on a **held-out set**, change exactly **one** variable (model OR
+tokenizer OR data OR decoding params), re-measure on the **same** set. Two changes at once and
+the delta is unattributable — and a regression can hide inside an improvement.

package/skills/shared/nlp/references/tokenization.md

@@ -0,0 +1,95 @@
+# Tokenization — mechanics, special tokens, fertility
+
+Depth overflow for SKILL.md section 1. Grounded in the HF NLP course (chapters 6.4–6.8) and the
+`tokenizers` library; verify current API at author time (transformers current major ~v5).
+
+## The four subword algorithms
+
+All three "learned" algorithms share a preprocessing step (normalization + pre-tokenization);
+they differ in how they build and apply the vocabulary.
+
+### BPE (Byte-Pair Encoding) — GPT-2, RoBERTa, and many others
+
+1. Start from a base vocabulary (characters, or the 256 bytes for byte-level).
+2. Count all adjacent token pairs in the corpus; **merge the single most frequent pair** into a
+   new token; record the merge rule.
+3. Repeat until the vocab reaches the target size. The output is an ordered list of merge rules.
+
+To tokenize new text: split into base units, then **replay the merge rules in learned order**.
+
+**Byte-level BPE** (GPT-2, RoBERTa) runs BPE over raw **bytes** instead of Unicode characters:
+
+- Base vocabulary is exactly **256** (every byte value), so it stays tiny.
+- **No `[UNK]` token is ever needed** — every possible character maps to some byte sequence, so
+  emoji and unseen scripts are always representable (HF NLP course ch.6.5). This is the key
+  robustness win over character/word tokenizers.
+
+### WordPiece — BERT, DistilBERT, ELECTRA
+
+Similar merge-based training, but instead of picking the most *frequent* pair it picks the pair
+that maximizes a **likelihood score**: `score = freq(pair) / (freq(first) × freq(second))`. This
+favors merging pieces whose parts are individually rare — it prefers to build up rare tokens.
+
+To tokenize: **greedy longest-match from the front** of each word. Continuation subwords are
+prefixed with `##` (e.g. `token`, `##izers`). If no subword matches, the whole word becomes
+`[UNK]` — so WordPiece *can* lose OOV content, unlike byte-level BPE.
+
+### Unigram — T5, ALBERT, mT5, XLNet (usually via SentencePiece)
+
+Works **top-down**: start from a large candidate vocabulary and iteratively **remove** the tokens
+whose removal least increases the total corpus negative-log-likelihood, until the target size is
+reached. Each remaining token keeps a probability/score.
+
+To tokenize: choose the **segmentation with the highest product of token probabilities**
+(Viterbi over candidate splits), not a left-to-right greedy pass.
+
+### SentencePiece — the wrapper, not a fourth algorithm
+
+SentencePiece is a *framework* (usually running Unigram, sometimes BPE) that treats the input as
+a raw stream and encodes spaces as a normal meta-symbol `▁`. Consequences:
+
+- **Language-agnostic** — no whitespace pre-tokenizer, so it works on languages that don't
+  space-separate words (Chinese, Japanese, Thai).
+- **Reversible** — `decode` is literally "concatenate tokens and swap `▁`→space," so
+  `decode(encode(x)) == x` with no detokenization rules. This is why multilingual models favor it.
+
+## Special tokens (names differ by family — never hand-roll)
+
+| Purpose | BERT/WordPiece | RoBERTa/GPT byte-level | T5/SentencePiece |
+|---|---|---|---|
+| Start / pooled repr | `[CLS]` | `<s>` | (uses `pad` as decoder start) |
+| Separator / end | `[SEP]` | `</s>` | `</s>` |
+| Padding | `[PAD]` | `<pad>` | `<pad>` |
+| Unknown fallback | `[UNK]` | `<unk>` (rare, byte-level) | `<unk>` |
+| Mask (MLM only) | `[MASK]` | `<mask>` | — |
+
+- `[CLS]`/`<s>` position holds the sentence-level vector used by a classification head.
+- `[PAD]` tokens must be excluded from attention via the `attention_mask` the tokenizer returns;
+  forgetting this lets padding leak into the representation.
+- Add task-specific special tokens with `tokenizer.add_special_tokens(...)` **and** call
+  `model.resize_token_embeddings(len(tokenizer))`, or the new ids index past the embedding table.
+
+## Offset mapping (needed for NER / span tasks)
+
+`return_offsets_mapping=True` returns `(char_start, char_end)` per token so you can map predicted
+token spans back to characters in the original string — essential for entity extraction and for
+aligning B-/I- labels to subwords.
+
+```python
+enc = tok("Ada Lovelace", return_offsets_mapping=True)
+# offsets like [(0,0),(0,3),(4,12),(0,0)] — (0,0) marks added special tokens
+```
+
+## Measure fertility before committing (multilingual)
+
+Fertility = mean tokens per word. Compare it across your languages on a parallel sample; a large
+gap is the unfairness/cost tax from SKILL.md section 4.
+
+```python
+def fertility(texts, tokenizer):
+    toks = sum(len(tokenizer.tokenize(t)) for t in texts)
+    words = sum(len(t.split()) for t in texts)
+    return toks / words          # ~1.1–1.4 English on a good tokenizer; much higher off-target
+
+# Compare the SAME sentences across languages; a 2–15× gap = over-fragmentation of that script.
+```

package/skills/shared/no-code-app/SKILL.md

@@ -0,0 +1,145 @@
+---
+name: no-code-app
+description: "Use when building an app on a no-code/low-code platform (Bubble, Softr, Glide over Airtable or Xano) — picking the platform, modeling the data, keeping usage-metered cost predictable, and planning the exit before lock-in. NOT a hand-written coded app (that is `nextjs`), NOT wiring SaaS tools together with no app UI (that is `automation-flows`)."
+tags: [no-code, low-code, bubble, glide, softr, app-builder, mvp]
+recommends: [nextjs, shopify, wordpress, automation-flows, spreadsheet-ops, notion-connector, stripe]
+origin: ai-hub
+---
+
+# No-code app
+
+You are deciding whether to build an app on a visual platform, which platform, how to model it, and where the wall is. No-code is not "the easy way" — it is a trade: you buy speed-to-first-user and pay it back in logic ceilings and metered cost. Your job is to make that trade with eyes open.
+
+**Decide the exit before you build the entrance.** If you cannot describe how you would leave, you are not building an MVP — you are building a hostage.
+
+No-code wins on **speed-to-first-user** — a working app in days, no deploy pipeline. That is the only thing it reliably wins. It dies on **deep conditional logic** and on **scale economics**: many-branch workflows get unmaintainable, and metering turns a $29 bill into a $400 one without warning.
+
+## Step 0: is no-code even the right tool?
+
+Run this gate before touching any builder. If any line is a clear yes, route out — do not rationalize staying.
+
+| If the user really wants... | Stop and go to |
+| --- | --- |
+| A hand-written app, full control over code | [../nextjs/SKILL.md](../nextjs/SKILL.md) (or [../react/SKILL.md](../react/SKILL.md)) |
+| A commerce storefront, products + checkout | [../shopify/SKILL.md](../shopify/SKILL.md) |
+| A content site / blog / CMS | [../wordpress/SKILL.md](../wordpress/SKILL.md) |
+| To wire SaaS tools together, no app UI | [../automation-flows/SKILL.md](../automation-flows/SKILL.md) |
+| A spreadsheet as the deliverable itself | [../spreadsheet-ops/SKILL.md](../spreadsheet-ops/SKILL.md) |
+| A Notion workspace, not a published app | [../notion-connector/SKILL.md](../notion-connector/SKILL.md) |
+
+No-code is the right tool when ALL hold: there is a real app UI (screens + data + logic), the logic is mostly CRUD + a handful of rules, time-to-launch matters more than scale today, and you accept the platform owns your runtime. If logic is the heart of the product (pricing engine, matching algorithm, real-time anything), no-code is wrong even if the UI is simple — write code.
+
+## Pick the platform
+
+Match the platform to the **app shape**, then to the **billing model** you will have to defend in six months. Why this order: the billing model, not the feature list, is what kills no-code projects.
+
+| App shape | Platform | Bills on | Why |
+| --- | --- | --- | --- |
+| Internal tool / client portal over existing data | **Softr** | Flat rate per workspace | No per-write metering, no charge for internal editors → predictable as you grow |
+| Customer-facing, simple, mobile-feel | **Glide** | "Updates" (data writes) | Fast to ship; cost scales with write volume, and rows are capped per plan (a standard source tops out far below Glide's own tables) — watch both |
+| Complex custom SaaS, heavy logic, custom workflows | **Bubble** | Workload Units (WU) | Most powerful builder; every query/workflow/API call burns WU, and tier allotments are small (Starter ~175K WU, Growth ~250K — only ~1.4x, not the 4x some stale write-ups claim) → cost is a function of usage, not seats |
+
+Rules:
+- **Default to Softr for anything that is "a nice front-end over a table."** Flat-rate billing means a usage spike does not become a surprise invoice.
+- **Reach for Bubble only when the logic genuinely needs it.** Bubble can do almost anything; that power is exactly why its WU bill is hard to predict. If you pick Bubble for a simple portal you are overpaying in money and complexity.
+- **If you find yourself fighting the builder to express logic, that is the signal to stop** and reconsider [../nextjs/SKILL.md](../nextjs/SKILL.md). The builder fighting back is the migration cliff sending a postcard.
+
+Exact tiers and numbers change — see [references/platform-limits.md](references/platform-limits.md). Do not hardcode prices into your design; design against the *metering model*.
+
+## Model the data first
+
+Tables and relations come before a single screen. Why: screens are cheap to rebuild, a wrong data model poisons every workflow and every cost estimate on top of it.
+
+Pick the backend by **logic complexity**, not by familiarity:
+
+| Backend | What it is | Use when | Watch out |
+| --- | --- | --- | --- |
+| Platform-native DB | Built into Bubble/Glide/Softr | Simple apps, one source of truth | Locked to the platform |
+| Airtable | Relational DB with a spreadsheet UI | Simple read/create, admin data entry | ~50K records/base cap, slow API |
+| Xano | Full backend (PostgreSQL + visual API builder) | Complex logic, performance, security | More setup; it is a real backend |
+
+The hybrid pattern that scales: **Airtable for human data entry/admin, Xano for API logic + performance + security.**
+
+Field-type and relation discipline:
+- Normalize. One entity = one table. A `bookings` row links to `customers` and `classes` by relation, it does not copy the customer's name and email into every booking.
+- Give every field its real type (date, number, single-select), not "text for everything." Wrong types make filters and aggregations expensive or impossible later.
+- Watch the **50K-record/base cap on Airtable** and the fact that on Bubble every relation traversal in a workflow is WU you pay for.
+
+Bad → Good:
+
+```text
+Bad  — one giant "Records" table, 40 columns, status in a text field,
+       customer name + email + class title all duplicated per row.
+       => filtering is slow, a rename means editing every row, WU/updates balloon.
+
+Good — customers | classes | bookings(customer→, class→, date, status:single-select)
+       => rename once, filter by relation, aggregate cleanly, predictable cost.
+```
+
+Sketch the model in plain text before you build it:
+
+```yaml
+customers: { id, name, email, phone }
+classes:   { id, title, starts_at, capacity, instructor }
+bookings:  { id, customer: ->customers, class: ->classes, status: [pending|paid|cancelled], created_at }
+payments:  { id, booking: ->bookings, amount, provider_ref }   # via stripe
+```
+
+Payments in any of these flows: see [../stripe/SKILL.md](../stripe/SKILL.md).
+
+## Build logic without footguns
+
+The metering model punishes the obvious-but-wrong way of doing things. Build defensively.
+
+- **Keep workflows shallow.** A 6-level nested conditional in a visual editor is unreadable and un-debuggable. If logic gets that deep, it belongs in code (Xano function or a real backend), not in the builder canvas.
+- **Batch writes; never loop an API call per row.** A workflow that fires one external API call per record across 2,000 rows is 2,000 metered operations — this is the #1 cause of an exploded Bubble WU bill and Glide update overage. Batch, or move the loop server-side.
+- **Validate server-side, not just in the UI.** Client-side-only validation in a no-code app is trivially bypassed; enforce rules in the backend (Bubble backend workflow, Xano endpoint) and gate writes with privacy/permission rules.
+- **Push expensive aggregation to scheduled jobs**, not to every page load. Recomputing a dashboard on each view multiplies reads by traffic.
+
+Bad → Good:
+
+```text
+Bad  — On booking confirmed: loop over all 2,000 customers, call email API per row.
+       => 2,000 metered ops every run; WU/update bill spikes; rate limits hit.
+
+Good — On booking confirmed: write one row; a scheduled batch job sends queued emails.
+       => one metered write per booking; sending is amortized and rate-safe.
+```
+
+## Know your limits and the cost ceiling
+
+The full 2026 tier/metering/record-cap table and worked cost examples live in [references/platform-limits.md](references/platform-limits.md). Operating rules:
+
+- **Estimate cost from the metering model, not the sticker price.** Bubble: count WU ≈ queries + workflow steps + API calls per user action, times expected actions/month. Glide: count data writes (updates) per user per month. A free or Starter tier that looks fine at demo traffic is the trap.
+- **Load-test at 10x expected rows, early.** A Bubble app that is snappy at 100 records can crawl at 10,000. Find the wall before your users do, not after.
+- **Know the hard caps**: Airtable ~50K records/base; per-tier app/user/row limits on every platform. Design so you do not silently hit one in month three.
+- **Watch the overage meters**: Bubble warns by email at 75% and 100% of your WU allotment, then charges overage. Wire those warnings to a human who can react.
+
+## Plan the exit
+
+This is the section everyone skips and everyone regrets. Do it on day 1.
+
+1. **Export your data on day 1 and on a schedule.** Confirm the platform actually lets you export records (~68% give you no code export — data export is the minimum you must verify).
+2. **Keep the data model portable.** Clean tables + relations (the model above) map onto Postgres/Prisma directly. A giant denormalized table does not.
+3. **Know your migrate-off triggers** and watch for them:
+   - Logic outgrows the canvas — many-branch conditionals, custom algorithms.
+   - Performance — it crawls at real row counts (the 10K-row test).
+   - Cost — the metered bill crosses what a real backend + hosting would cost.
+4. **Price the rewrite honestly.** ~25-30% of no-code projects get rewritten when they outgrow the tool, at $50K-$250K. Knowing that number is what lets you decide *now* whether to start in code via [../nextjs/SKILL.md](../nextjs/SKILL.md) instead.
+
+When two of those triggers fire, stop adding features and start the migration plan — do not keep pouring work into a runtime you are about to leave.
+
+## Anti-patterns
+
+| Bad | Good | Why |
+| --- | --- | --- |
+| Reaching for Bubble because "no-code is faster" | Run Step 0 gate first | Coded apps/stores/blogs route to nextjs/shopify/wordpress |
+| Building screens, then bolting on data | Model tables + relations first | Wrong model poisons every workflow and cost estimate |
+| One giant table, 40 columns, text-typed everything | Normalized tables linked by relation | Cheap renames, fast filters, predictable metering |
+| Airtable for complex logic + scale | Xano (or hybrid: Airtable entry + Xano logic) | Airtable ~50K cap + slow API; Xano is a real backend |
+| Per-row API call in a workflow loop | Batch writes / scheduled jobs | #1 cause of WU/update bill explosions |
+| Client-side-only validation | Enforce rules server-side + permission gates | UI validation is trivially bypassed |
+| Pricing off the monthly sticker | Estimate from the metering model | WU/updates scale with usage, not the headline price |
+| Testing only at demo scale (100 rows) | Load-test at 10x expected rows early | Snappy at 100 can crawl at 10,000 |
+| No export plan, hope you never leave | Export day 1, keep model portable | 68% offer no code export; rewrite is $50K-$250K |
+| Picking Bubble for a simple internal portal | Softr (flat-rate) for portals over data | Predictable cost, no per-write surprise |

package/skills/shared/no-code-app/evals/README.md

@@ -0,0 +1,3 @@
+# Evals: no-code-app
+
+This is a process/judgment skill — there is no compiled artifact, so there is no `verify.sh` and no automated runner. Evaluate by hand: read `cases.yaml` against `SKILL.md`. For each `should_trigger` prompt, confirm the description's "Use when…/Triggers:" line would plausibly route an agent here; for each `should_not_trigger`, confirm the named `route_to` sibling is the better home and that this skill's boundary (`NOT a coded web app (that is nextjs)`) sends it away. The `capability` case is a rubric check: an agent following the skill on the yoga-studio brief should produce all five `must_include` items — Step 0 gate, a platform pick justified by app shape + billing model, a normalized data model with a named backend, a metered cost/limit estimate with a load-test note, and an exit/export plan. If any are missing, the SKILL.md body needs tightening.