rsc-universal 0.1.1

package/skills/email-deliverability/SKILL.md

@@ -0,0 +1,168 @@
+---
+name: email-deliverability
+description: "Use when mail you send lands in spam, gets rejected, or fails the Gmail/Yahoo bulk-sender checks, and you must fix authentication and reputation rather than the sending code — setting SPF/DKIM/DMARC DNS records so they align, raising a cold domain through warmup, keeping the spam-complaint rate under threshold, adding one-click unsubscribe, getting the brand logo and verified checkmark to show via BIMI, and reading why 'soft' rejections and Postmaster Tools say you are throttled. Triggers: 'our emails go to spam since the Gmail update', 'DMARC fails even though SPF passes', 'how do I warm up a new sending domain', 'do I need one-click unsubscribe to send to gmail', '550 5.7.26 unauthenticated', 'how do I get our logo and the blue checkmark to show in Gmail with BIMI', 'los correos caen en spam y el SPF está bien', 'configurar BIMI i el logo verificat al correu'. NOT putting transactional/bulk email on the wire through a provider API (that is email-connector)."
+tags: [email, deliverability, spf, dkim, dmarc, bimi, sender-reputation, bulk-sender]
+recommends: [email-connector, newsletter, cold-outreach, gdpr-privacy, data-policy]
+origin: risco
+---
+
+# email-deliverability — make mail authenticate and land in the inbox
+
+This skill owns one layer: **why mail you already send gets filtered, deferred,
+or rejected, and the DNS + reputation work that fixes it.** It does not put mail
+on the wire — that is `email-connector`. It starts where the send technically
+succeeds but the message never reaches the inbox.
+
+The lever is almost never the sending code. It is three DNS records that must
+*align*, a complaint rate you must hold down, and a reputation you build slowly
+and lose fast. Treat this as ops on a domain, not a code change.
+
+## When to use / When NOT to use
+
+**Use when:** mail lands in spam or Promotions and you suspect auth/reputation,
+not copy; a receiver returns `550 5.7.26` (unauthenticated), `421`/`451`
+deferrals, or `dmarc=fail`; you must meet the Gmail/Yahoo bulk-sender rules
+before a launch; you are standing up a brand-new sending domain and need a
+warmup plan; Google Postmaster Tools shows a rising spam rate or "Bad" domain
+reputation; you want the brand logo and a verified checkmark to show in Gmail
+(BIMI + VMC/CMC).
+
+**Do NOT use for:**
+
+- **Sending the mail at all** — provider SDK, `sendEmail()` seam, templates,
+  retries, batch caps, bounce/complaint webhooks → `email-connector`. That skill
+  feeds the suppression list; this one explains the reputation it protects.
+- **Subject lines, open/click optimization, list growth** → `newsletter`.
+- **Outbound prospecting sequences, lead lists** → `cold-outreach` (deliverability
+  is a *constraint* on it, not the same job).
+- **Consent, lawful basis, retention of the address list** → `gdpr-privacy` and
+  `data-policy`. Unsubscribe *mechanics* live here; consent *law* does not.
+
+## The three records that must align
+
+SPF, DKIM, and DMARC are not three ways to do the same thing. DMARC is the
+policy; it *passes* only when SPF or DKIM both authenticate **and align** with
+the visible `From:` domain. Most "SPF is set but DMARC fails" tickets are an
+alignment miss, not a missing record.
+
+| Record | What it asserts | Common break |
+|---|---|---|
+| SPF (TXT) | This IP/host is allowed to send for the domain | Lists the provider but the `Return-Path` is a different domain → no alignment |
+| DKIM (TXT) | This message body+headers are signed by a key the domain published | Provider signs with `provider.net`, not your domain → no alignment; or a deprecated 1024-bit key (publish 2048-bit, rotate periodically) |
+| DMARC (TXT at `_dmarc`) | What to do when neither aligns, plus where to send reports | Policy left at `p=none` forever; never tightened |
+
+```dns
+; DMARC — start at none to collect reports, then tighten to quarantine/reject.
+_dmarc.acme.com.  IN TXT  "v=DMARC1; p=none; rua=mailto:dmarc@acme.com; fo=1"
+```
+
+Alignment rule, stated once: the domain in `From:` must match the domain SPF
+authenticated (the `Return-Path`/envelope domain) **or** the domain in the DKIM
+`d=` tag. Use a custom Return-Path and a domain-keyed DKIM selector through your
+provider so at least one aligns. Verifying SPF alone passing is the classic
+false comfort.
+
+## Gmail / Yahoo bulk-sender rules
+
+If you send **more than ~5,000 messages/day** to Gmail or Yahoo accounts, these
+are enforced, not advisory. Enforcement ramped from late 2025; failures now draw
+temporary and permanent rejections, not silent spam-foldering.
+
+1. **SPF + DKIM both set, and DMARC present** at minimum `p=none`. At least one
+   of SPF/DKIM must align with `From:`.
+2. **One-click unsubscribe** (`List-Unsubscribe` + `List-Unsubscribe-Post`) on
+   marketing/promotional mail, honored within **2 days**. Transactional mail
+   (password reset, receipt, shipping) is exempt.
+3. **Spam-complaint rate** kept low. The hard threshold where filtering kicks in
+   is **0.3%** (Postmaster Tools); Google's reliable-inbox target is **below
+   0.1%**. Treat 0.3% as the cliff, 0.1% as the speed limit.
+
+```text
+List-Unsubscribe: <https://acme.com/u/abc123>, <mailto:unsub@acme.com?subject=unsub>
+List-Unsubscribe-Post: List-Unsubscribe=One-Click
+```
+
+## BIMI — the verified logo, last
+
+BIMI shows your brand logo (and on Gmail a blue verified checkmark) next to the
+message. It is the **last** step, never a fix for placement: it requires you have
+*already* passed DMARC at enforcement, and it changes how a delivered message
+looks, not whether it gets delivered.
+
+Hard prerequisite: DMARC at `p=quarantine` or `p=reject` with `pct=100`. A record
+left at `p=none` does **not** qualify — that is why "add BIMI to escape the spam
+folder" is backwards; you cannot publish it until the auth work is done.
+
+Whether the logo (and checkmark) actually render depends on the certificate:
+
+| Approach | Cost / proof | Where it shows |
+|---|---|---|
+| Self-asserted (no certificate) | Free; logo only | Yahoo, Fastmail — **not** Gmail |
+| CMC (Common Mark Certificate) | Cheaper; ~12 months of public logo use, no registered trademark | Gmail + the rest, with checkmark |
+| VMC (Verified Mark Certificate) | ~$749–$1,500/yr; needs a registered (or government-modified) trademark | Gmail + the rest, with checkmark |
+
+Active mark-certificate issuers in 2026: DigiCert, GlobalSign, SSL.com (Sectigo
+resells; Entrust exited mark certificates in 2025). The logo file must be **SVG
+Tiny PS 1.2**, square, with no scripts or external references.
+
+```dns
+; BIMI — only valid once DMARC is at quarantine/reject with pct=100.
+; a= is the VMC/CMC PEM; omit it for a self-asserted (non-Gmail) logo.
+default._bimi.acme.com.  IN TXT  "v=BIMI1; l=https://acme.com/logo.svg; a=https://acme.com/vmc.pem"
+```
+
+Do BIMI only after weeks of clean, enforced DMARC. It amplifies a good
+reputation; it does not create one.
+
+## Warm a cold domain (decision flow)
+
+A brand-new domain has zero reputation; blasting volume on day one looks exactly
+like a spammer. Warm only if the domain genuinely has no history.
+
+- **New domain, no send history** → warm: start ~20-50/day to *engaged*
+  recipients, roughly double every few days over 4-6 weeks, watch Postmaster
+  reputation before each step up. Send your most-opened content first.
+- **Established domain, new provider/IP** → warm the *IP* path, not the domain;
+  migrate a slice of volume and ramp.
+- **Established domain, same IP, sudden spam folder** → do NOT warm; this is a
+  reputation or auth regression. Read the rejection code and Postmaster trend
+  first (see the error table).
+
+Skipping warmup to "just send the campaign" is the most common self-inflicted
+blacklisting. There is no fast path; reputation is earned by consistent,
+low-complaint sending.
+
+## Read the rejection, then act
+
+| Signal | Likely cause | Fix |
+|---|---|---|
+| `550 5.7.26` | Message unauthenticated — SPF and DKIM both failed/missing | Publish SPF + DKIM through the provider; confirm at least one aligns |
+| `dmarc=fail` in headers, SPF=pass | Alignment miss: `Return-Path` ≠ `From:` domain | Set a custom Return-Path on your domain, or align the DKIM `d=` |
+| `421`/`451` deferral, then later delivery | Receiver throttling an unwarmed/spiky sender | Slow the ramp; spread volume; warm the domain |
+| Postmaster "Bad" domain reputation | Sustained complaints / spam-trap hits | Cut volume, prune unengaged, fix consent, hold under 0.1% complaints |
+| Lands in Promotions (not Spam) | Not a failure — bulk/marketing classification | Leave it; do not chase the Primary tab by faking transactional headers |
+
+## Anti-patterns
+
+| Anti-pattern | Why it breaks | Do instead |
+|---|---|---|
+| "SPF passes, so we're authenticated" | DMARC needs *alignment*; SPF on a mismatched Return-Path still fails DMARC | Verify DMARC result in headers, not SPF alone; align Return-Path or DKIM |
+| Leaving DMARC at `p=none` forever | Publishes intent to enforce nothing; spoofers and filters both notice | Collect `rua` reports, then move to `quarantine` then `reject` |
+| `p=reject` on day one with no report review | Silently drops your own legitimate sub-streams (CRM, support tools) | Stage `none` → `quarantine` → `reject`, reading reports at each step |
+| Blasting full volume from a new domain | Looks like spam; gets throttled/blacklisted with no recovery for weeks | Warm: tiny start to engaged users, ramp over 4-6 weeks |
+| Buying/scraping a list to hit volume | Spam traps + complaints spike past 0.3% → domain reputation tanks | Send only to opted-in, engaged recipients; prune the unengaged |
+| Faking transactional headers to dodge Promotions | Violates bulk rules; risks rejection, not just foldering | Accept Promotions placement; earn Primary via engagement |
+| Routing the unsubscribe to a form that takes a week | Breaks the 2-day one-click honor rule; raises complaints | Honor `List-Unsubscribe-Post` one-click within 2 days, automatically |
+| Adding BIMI to fix spam-foldering | BIMI needs DMARC enforcement you do not have yet, and it changes logo display, not placement | Fix auth + reputation first; add BIMI last as a branding layer |
+
+## See also
+
+- `../email-connector/SKILL.md` — the provider/SDK send path, suppression list,
+  and bounce/complaint webhook that this skill's reputation rules sit on top of.
+- `../secure-coding/SKILL.md` — handling the DKIM private key and provider API
+  secrets without leaking them.
+
+Recommended companions (siblings): `email-connector` for the actual send,
+`newsletter` for the copy/engagement side that drives complaint rate down,
+`cold-outreach` (deliverability gates it), and `gdpr-privacy` / `data-policy`
+for the consent and retention behind a clean list.

package/skills/email-deliverability/evals/README.md

@@ -0,0 +1,21 @@
+# Evals — email-deliverability
+
+`cases.yaml` holds three blocks. `should_trigger` (7 prompts) are situations this
+skill must own — the SPF-passes-but-DMARC-fails alignment trap, domain warmup,
+the one-click-unsubscribe rule, the BIMI verified-logo request, and a Spanish
+phrasing of the spam-folder flagship. `should_not_trigger` (5 prompts) are
+near-misses that each route to a
+real sibling (`email-connector`, `newsletter`, `cold-outreach`, `gdpr-privacy`,
+`data-policy`) so the boundary is checked, not just recall — the sharpest being
+the send-path question that belongs to `email-connector`. `capability` (1
+scenario) is an 8k/day Gmail-sender remediation with a `must_include` rubric
+covering alignment diagnosis, SPF/DKIM fix, the bulk-sender threshold, one-click
+unsubscribe, complaint-rate targets, and staged DMARC tightening.
+
+There is no automated runner. To evaluate, paste each `should_trigger` /
+`should_not_trigger` prompt into an agent loaded with the catalog and confirm it
+selects (or declines) `email-deliverability` and routes to the named sibling.
+For `capability`, run the scenario and check the answer against every
+`must_include` line — then run `scripts/verify.sh <path-to-dns-or-config>` to
+mechanically confirm the DMARC policy and authentication invariants in any DNS
+zone / mail config the answer produces.

package/skills/email-deliverability/evals/cases.yaml

@@ -0,0 +1,45 @@
+skill: email-deliverability
+
+should_trigger:
+  - prompt: "Our transactional emails started going to spam in Gmail right after their big update. SPF is set. What's wrong?"
+    why: "Flagship symptom: inbox placement regression tied to the Gmail bulk-sender enforcement, with the SPF-passes-but-still-spam alignment trap."
+  - prompt: "DMARC keeps failing in the headers even though SPF passes — how do I make them agree?"
+    why: "Non-obvious core: DMARC fails on alignment, not on a missing SPF record. The fix is Return-Path / DKIM d= alignment, the skill's central distinction."
+  - prompt: "We just bought a fresh domain for outreach. How do I warm it up so we don't get blacklisted on the first send?"
+    why: "Warmup is a named decision flow here. Distinguishes new-domain (warm) from same-IP-regression (don't warm)."
+  - prompt: "Do we legally need one-click unsubscribe to keep sending marketing mail to gmail addresses, and how fast must we honor it?"
+    why: "Bulk-sender requirement: List-Unsubscribe-Post one-click honored within 2 days; transactional exempt. Specific rule the skill owns."
+  - prompt: "Los correos nos caen en spam y el SPF está bien configurado, ¿qué revisamos?"
+    why: "Spanish phrasing of the flagship alignment problem — must fire across language, routes to DMARC alignment + reputation checks."
+  - prompt: "Postmaster Tools shows our domain reputation dropped to Bad and we're getting 421 deferrals. What's the recovery path?"
+    why: "Reputation + deferral diagnosis. Non-obvious that 421 is throttling and the fix is volume/complaint reduction, not resending harder."
+  - prompt: "How do I get our company logo and that blue checkmark to show up next to our emails in Gmail?"
+    why: "BIMI/VMC is in scope. Non-obvious that the logo depends on DMARC already being at enforcement and on a VMC/CMC certificate, not on a copy or sending change."
+
+should_not_trigger:
+  - prompt: "Wire up Resend in our Node app with a sendEmail() seam and idempotent retries for password-reset mail."
+    route_to: "email-connector"
+    why: "Putting mail on the wire through a provider SDK — the send path, not the reputation behind it. email-connector owns the transport and suppression list."
+  - prompt: "Write me three subject-line variants and an A/B plan to lift open rates on our weekly digest."
+    route_to: "newsletter"
+    why: "Copy and engagement optimization, not authentication or reputation DNS. newsletter owns subject/open/click."
+  - prompt: "Build a 6-step cold outreach sequence with personalized first lines for this lead list."
+    route_to: "cold-outreach"
+    why: "Prospecting sequence design. Deliverability constrains it but the deliverable here is the sequence, which is cold-outreach."
+  - prompt: "Do we have lawful basis under GDPR to email this list, and how long can we retain the addresses?"
+    route_to: "gdpr-privacy"
+    why: "Consent and retention law, not unsubscribe mechanics or DNS. The legal basis behind the list is gdpr-privacy."
+  - prompt: "Define our company-wide policy for how long we keep customer contact data and who can access it."
+    route_to: "data-policy"
+    why: "Data governance / retention policy as the deliverable, not inbox placement. data-policy, not deliverability."
+
+capability:
+  - scenario: "A SaaS sends ~8,000 marketing emails/day to mostly Gmail recipients. They report messages now bounce with 550 5.7.26 or land in spam; SPF passes but headers show dmarc=fail. Diagnose and give a concrete remediation plan."
+    must_include:
+      - "Identify the root cause as a DMARC alignment failure (SPF passing on a Return-Path that does not match the From: domain), not a missing record"
+      - "Prescribe aligning at least one of SPF (custom Return-Path on the sending domain) or DKIM (domain-keyed d= selector via the provider)"
+      - "Note the 5,000/day Gmail-Yahoo bulk-sender threshold applies: SPF + DKIM + DMARC at minimum p=none, with at least one aligned"
+      - "Require one-click unsubscribe (List-Unsubscribe + List-Unsubscribe-Post) honored within 2 days on the marketing mail"
+      - "Set a spam-complaint target below 0.1% with 0.3% as the enforcement cliff, watched in Postmaster Tools"
+      - "Stage DMARC policy none -> quarantine -> reject while reading rua reports, rather than jumping straight to reject"
+      - "Treat BIMI as out of scope for this fix: it requires DMARC already at enforcement and does not change placement, so do not propose it as the spam remedy"

package/skills/email-deliverability/scripts/verify.sh

@@ -0,0 +1,98 @@
+#!/usr/bin/env bash
+#
+# verify.sh — DNS / mail-config deliverability linter. NO network, NO DNS
+# lookups. Scans a target dir or file for the authentication anti-patterns this
+# skill bans and exits non-zero on a hard finding.
+#
+# Read-only. Never writes, edits, or sends anything.
+#
+# Checks (text-only; this does not resolve DNS):
+#   1. A DMARC record left at "p=none" — a WARN, not a fail: fine for the report
+#      phase, but a reminder it must be tightened to quarantine/reject.
+#   2. A DMARC record at "p=reject" with NO rua= reporting address — a FAIL:
+#      enforcing with no visibility silently drops legitimate sub-streams.
+#   3. A "v=spf1" record ending in "+all" — a FAIL: passes any sender, defeats SPF.
+#
+# Usage:
+#   scripts/verify.sh [PATH ...]    PATH = a zone/config file or dir; default "."
+#
+# Exit 0 = no hard finding (clean/empty target passes). 1 = a hard finding.
+# Exit 2 = usage/environment problem.
+
+set -euo pipefail
+
+# Never let this script flag itself. grep -rn emits "<path>:<line>:..."; strip
+# any hit whose path component basenames to this script (relative or absolute).
+SELF_BASE="$(basename "${BASH_SOURCE[0]}")"
+drop_self() {
+  awk -v self="$SELF_BASE" -F: '{
+    p = $1
+    n = split(p, parts, "/")
+    if (parts[n] != self) print
+  }'
+}
+
+if [ "$#" -eq 0 ]; then
+  set -- "."
+fi
+
+declare -a TARGETS=()
+for t in "$@"; do
+  if [ -e "$t" ]; then
+    TARGETS+=("$t")
+  else
+    echo "verify.sh: no such path: $t" >&2
+  fi
+done
+
+if [ "${#TARGETS[@]}" -eq 0 ]; then
+  echo "verify.sh: nothing to scan"
+  exit 0
+fi
+
+INCLUDE=(--include='*.txt' --include='*.zone' --include='*.dns'
+  --include='*.conf' --include='*.cf' --include='*.yaml' --include='*.yml'
+  --include='*.tf' --include='*.json' --include='*.env')
+EXCLUDE=(--exclude-dir=.git --exclude-dir=node_modules --exclude=verify.sh)
+
+fail=0
+emit() {
+  echo "FAIL  $1"
+  while IFS= read -r line; do
+    [ -n "$line" ] && echo "        $line"
+  done <<< "$2"
+  fail=1
+}
+
+# --- check 1: DMARC p=none (WARN) -----------------------------------------
+none="$( { grep -rnI "${EXCLUDE[@]}" "${INCLUDE[@]}" \
+  -E 'v=DMARC1[^"]*p=none' "${TARGETS[@]}" 2>/dev/null || true; } | drop_self)"
+if [ -n "${none//[$'\n']/}" ]; then
+  echo "WARN  DMARC at p=none (report phase only — tighten to quarantine/reject):"
+  while IFS= read -r l; do [ -n "$l" ] && echo "        $l"; done <<< "$none"
+fi
+
+# --- check 2: DMARC p=reject with no rua= reporting (FAIL) -----------------
+reject_hits=""
+while IFS= read -r l; do
+  [ -z "$l" ] && continue
+  if ! printf '%s' "$l" | grep -q 'rua='; then
+    reject_hits+="${l}"$'\n'
+  fi
+done < <( { grep -rnI "${EXCLUDE[@]}" "${INCLUDE[@]}" \
+  -E 'v=DMARC1[^"]*p=reject' "${TARGETS[@]}" 2>/dev/null || true; } | drop_self)
+if [ -n "${reject_hits//[$'\n']/}" ]; then
+  emit "DMARC p=reject with no rua= report address (enforcing blind drops legit mail)" "$reject_hits"
+fi
+
+# --- check 3: SPF +all (FAIL) ---------------------------------------------
+spf="$( { grep -rnI "${EXCLUDE[@]}" "${INCLUDE[@]}" \
+  -E 'v=spf1[^"]*\+all' "${TARGETS[@]}" 2>/dev/null || true; } | drop_self)"
+if [ -n "${spf//[$'\n']/}" ]; then
+  emit "SPF ends in +all (passes any sender — use -all or ~all)" "$spf"
+fi
+
+if [ "$fail" -eq 0 ]; then
+  echo "OK    no hard deliverability anti-pattern found"
+fi
+exit "$fail"

package/skills/embeddings-search/SKILL.md

@@ -0,0 +1,193 @@
+---
+name: embeddings-search
+description: "Use when results from a semantic search feel irrelevant, when you don't know which embedding model or chunk size to use, when you need to add a reranker or hybrid scoring, or when you have no way to tell whether a retrieval change actually helped. Symptoms: search returns garbage, exact-match queries miss, paraphrases work but keywords don't, recall looks fine but the top result is wrong, every chunking idea is a guess. Triggers: 'which embedding model and chunk size should I use', 'my semantic search returns irrelevant results', 'how do I measure if retrieval is good — recall@k, nDCG', 'add a reranker and hybrid BM25+vector', 'exact product codes return nothing but paraphrases work fine', 'recall@k is fine but the top answer is wrong', 'quin model d'embeddings i quina mida de chunk faig servir per a aquests documents', 'la cerca semàntica retorna resultats irrellevants'. NOT operating the vector store (that is vector-db)."
+tags: [embeddings, semantic-search, chunking, hybrid-search, reranking, rrf, retrieval-eval, mteb, matryoshka]
+recommends: [vector-db, rag, structured-extraction, prompt-engineering, postgresdb]
+origin: risco
+---
+
+# embeddings-search — make and judge the vectors
+
+You own the **embedding technique layer**: turn a corpus into searchable vectors, turn a
+question into a good retrieval, and **measure whether that retrieval is any good**. When the
+symptom is "results are irrelevant," "I don't know which model or chunk size," or "how do I
+even tell if search improved," this is the skill. You stop the moment the right chunks come
+back, measured by a number. You do not assemble a prompt or generate an answer.
+
+Route the adjacent surfaces away:
+
+- **Operating the store** — collection schema, HNSW/IVFFlat tuning, metadata-filter path,
+  quantization, `ef_search` recall knobs → [`../vector-db/SKILL.md`](../vector-db/SKILL.md).
+  You decide *what vectors go in and how to query*; vector-db decides *how the store holds and
+  serves them*.
+- **The full retrieve → rerank → prompt → generate → answer loop** and its groundedness /
+  faithfulness eval → [`../rag/SKILL.md`](../rag/SKILL.md).
+- **Pulling typed fields out of documents** (invoice number, date, total) → [`../structured-extraction/SKILL.md`](../structured-extraction/SKILL.md).
+- **Writing the prompt the model reasons with** → [`../prompt-engineering/SKILL.md`](../prompt-engineering/SKILL.md).
+
+## 1. Pick the embedding model
+
+Decide on three axes: language coverage, quality tier (read MTEB but don't worship it), and
+cost — where cost is set by **dimensions**, because dims set storage and memory.
+
+| Model | Best when | Dims (Matryoshka) | Max input | ~Price /1M tok | Query/doc asymmetry |
+|---|---|---|---|---|---|
+| OpenAI `text-embedding-3-small` | Cheap English/multi baseline | 1536 (truncatable) | 8191 tok | ~$0.02 | none required |
+| OpenAI `text-embedding-3-large` | Higher quality, still API-simple | 3072 (truncatable) | 8191 tok | ~$0.13 | none required |
+| Cohere `embed-v4` | Strong multilingual, API | up to 1536 | long | API-priced | `search_query` vs `search_document` |
+| Voyage `voyage-3-large` | Retrieval-specialised, top tasks | model-set | long | API-priced | yes (`input_type`) |
+| Gemini Embedding | Tops MTEB English retrieval (~68.3) | truncatable | long | API-priced | yes (task type) |
+| `BGE-M3` / `e5` (open) | Self-host, no per-token bill | 1024 (BGE-M3) | long | self-host | yes (`query:` / `passage:`) |
+
+Quality anchor (mid-2026 MTEB English retrieval): Gemini ~68.3, Cohere `embed-v4` ~65.2,
+OpenAI `3-large` ~64.6, `BGE-M3` ~63.0. MTEB is the standard comparison, not a verdict on
+*your* domain (see `references/models.md` for the over-trust caveat).
+
+Two hard rules — each is a **silent** failure, no error, just worse results:
+
+- **Match the distance metric to the model.** A cosine-trained model indexed or queried with
+  L2 ranks silently wrong. Cosine → `<=>` in pgvector / `Distance.COSINE` in Qdrant. The index
+  operator itself is vector-db's job; the *requirement* originates from the model, so state it
+  in your config.
+- **Respect query/document asymmetry.** Cohere, Voyage, Gemini, e5, BGE expect a different
+  prompt or `input_type` for the query vs the stored passage. Embed both sides identically and
+  recall silently drops.
+
+**Dimensions = cost.** A 1024-dim float32 vector is 4 KB; at 10M docs that is 40 GB, and
+doubling dims doubles storage and memory. Matryoshka-trained models (OpenAI 3-*, Cohere,
+Gemini) let you **truncate** dims for graceful degradation — never re-embed the whole corpus
+just to shrink vectors.
+
+## 2. Chunk the corpus
+
+Start boring. Upgrade only when a number tells you to.
+
+```python
+# Default recipe: recursive split, TOKEN-accurate count, 10–20% overlap.
+import tiktoken
+from langchain_text_splitters import RecursiveCharacterTextSplitter
+
+enc = tiktoken.get_encoding("cl100k_base")
+splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
+    encoding_name="cl100k_base",
+    chunk_size=512,      # tokens, not characters
+    chunk_overlap=64,    # ~12% — keeps sentences from being cut mid-thought
+)
+chunks = splitter.split_text(document_text)
+```
+
+Counting by characters instead of tokens is the most common own-goal: 512 characters is
+~100–130 tokens of English and far fewer of CJK, so "512" silently means different things per
+language and per model limit.
+
+Upgrade ladder — graduate only when retrieval metrics (section 6) justify the added compute:
+
+| Strategy | When it pays | Cost |
+|---|---|---|
+| Recursive (default) | Always start here | lowest |
+| Semantic (group by meaning) | Topic-mixed pages where fixed splits cut mid-idea; ~70% lift over naive in some benchmarks | one extra embed pass |
+| Late chunking | Docs heavy with pronouns/anaphora ("it", "the company"); +10–12% on those | needs a long-context model |
+| Contextual retrieval (prepend a heading/summary per chunk) | Chunks that aren't self-contained without their section | higher compute, more tokens stored |
+
+**Embed the searchable text; store the rest as metadata.** What you embed is what gets matched
+— don't bury the answer text under boilerplate, and don't embed raw HTML.
+
+## 3. Construct the query
+
+The query is half the retrieval. Embed it the way the model expects, then improve it only when
+recall data says you should.
+
+```python
+# Asymmetric model: query and document use DIFFERENT input_type. Getting this wrong is silent.
+q_vec   = embed(text=user_question, input_type="search_query")     # Cohere / Voyage
+d_vec   = embed(text=passage,       input_type="search_document")
+# e5 / BGE convention is a textual prefix instead:
+#   query    -> "query: how do refunds work"
+#   passage  -> "passage: Refunds are processed within 14 days…"
+```
+
+Query-side techniques, when each pays:
+
+- **Query rewriting** — when user queries are terse or full of pronouns; normalise before
+  embedding.
+- **HyDE** (embed a hypothetical answer, not the question) — when questions are short and
+  answers are long/technical, so the answer-shaped vector lands nearer the passage.
+- **Multi-query** (fan out 3–4 paraphrases, union the hits) — when one phrasing under-recalls;
+  costs N embeds and a dedup.
+
+## 4. Hybrid + rerank
+
+Dense and sparse fail in complementary ways: **BM25 nails exact terms, IDs, SKUs, rare
+tokens**; **dense nails paraphrase**. That is why exact-match queries return nothing while
+paraphrases work — the fix is adding sparse, not a bigger embedding model.
+
+Fuse by **rank, not score**, with Reciprocal Rank Fusion so you never have to calibrate BM25
+tf-idf magnitudes against cosine magnitudes per corpus:
+
+```python
+# RRF: each doc scores 1/(k + rank) summed across the dense and sparse lists. k≈60.
+def rrf(*ranked_lists, k=60):
+    scores = {}
+    for lst in ranked_lists:                 # fan-in 20–100 per list
+        for rank, doc_id in enumerate(lst):  # rank is 0-based
+            scores[doc_id] = scores.get(doc_id, 0) + 1.0 / (k + rank + 1)
+    return sorted(scores, key=scores.get, reverse=True)
+```
+
+Then a **cross-encoder reranker** sits AFTER fusion: take top-50, score each against the
+original query, keep top-5 for downstream use.
+
+- Current models: Cohere `rerank-v4.0-pro` / `rerank-v4.0-fast` (note `rerank-3.5` is
+  deprecated). Voyage `rerank-2.5` (2025-08-11) is the first widely available
+  **instruction-following** reranker — 32K-token context (8× Cohere v3.5), reports +7.94%
+  accuracy vs Cohere v3.5 on a 93-dataset suite.
+- **A reranker raises precision but cannot recover a doc the retriever never returned.** Recall
+  is upstream. If the right chunk isn't in the top-50, no reranker saves you — fix recall first.
+
+The fusion and index *mechanics per engine* (how Qdrant/Weaviate/pgvector run hybrid) are
+vector-db's: [`../vector-db/SKILL.md`](../vector-db/SKILL.md).
+
+## 5. Measure retrieval quality
+
+This is the rigor of the skill. "Search is bad" is not actionable; "recall@10 is 0.62" is.
+
+1. **Build a golden query set** — 30–50+ labeled `query → relevant doc ids` pairs drawn from
+   real questions. This is the asset; everything else is reproducible from it.
+2. **Pick metrics** (definitions + runnable skeleton in `references/evaluation.md`):
+   - **recall@k** — of the truly relevant docs, how many landed in the top-k. Catches "the
+     right chunk never came back."
+   - **nDCG@k** — rewards relevant docs ranked higher. Catches "right docs, wrong order."
+   - **MRR** — how high the *first* relevant doc sits. Catches "the one answer is buried."
+3. **Move-the-number loop.** Establish a baseline, **change exactly one thing** (model OR chunk
+   size OR fusion OR reranker), re-measure on the **same** query set. Two changes at once and
+   you learn nothing.
+
+A retrieval change shipped without a before/after number on a golden set is a guess. `verify.sh`
+flags hybrid/rerank artifacts that mention no recall/nDCG/MRR for exactly this reason.
+
+## Anti-patterns
+
+| Anti-pattern | Why it bites | Do instead |
+|---|---|---|
+| Chunk size in characters | "512" means a different token count per language/model | Token-accurate count (`tiktoken`/model tokenizer) |
+| Same `input_type` for query and document | Asymmetric models silently lose recall, no error | `search_query` vs `search_document` (or `query:`/`passage:`) |
+| Cosine model indexed/queried with L2 | Ranking is silently wrong | Match metric to model (cosine → `<=>`) |
+| Add a reranker to fix bad recall | Reranker only reorders what retrieval returned | Fix recall (hybrid, chunking, model) first |
+| No overlap on prose | Sentences cut mid-thought lose the answer | 10–20% overlap |
+| Tuning by eyeballing one query | One query isn't a measurement | Golden set + recall@k/nDCG before vs after |
+| Trusting MTEB rank for your domain | Leaderboard ≠ your corpus/language | Eval the top 2–3 on your own golden set |
+| Over-large dims "for safety" | Doubles storage/RAM, little recall gain | Right-size; truncate via Matryoshka |
+| Re-embedding the corpus to change dims | Wasteful when the model is Matryoshka-trained | Truncate dims, don't re-embed |
+| Embedding raw HTML/boilerplate | Match signal drowns in markup | Embed clean text; keep the rest as metadata |
+
+## References & siblings
+
+- `references/models.md` — current model matrix (dims, max tokens, price, `input_type`
+  convention, Matryoshka support) and how to read MTEB without over-trusting it.
+- `references/evaluation.md` — golden-set construction, recall@k / nDCG@k / MRR definitions, a
+  runnable Python eval skeleton, and the A/B-one-change methodology.
+
+Siblings: store ops → [`../vector-db/SKILL.md`](../vector-db/SKILL.md) · full answer loop →
+[`../rag/SKILL.md`](../rag/SKILL.md) · typed extraction →
+[`../structured-extraction/SKILL.md`](../structured-extraction/SKILL.md) · prompt design →
+[`../prompt-engineering/SKILL.md`](../prompt-engineering/SKILL.md).

package/skills/embeddings-search/evals/README.md

@@ -0,0 +1,10 @@
+# Evals — embeddings-search
+
+These cases are routing and capability checks for a human or an LLM grader, not an automated
+test suite. To run them: read each `should_trigger` prompt and confirm an agent would reach for
+`embeddings-search`; read each `should_not_trigger` prompt and confirm it routes to the named
+sibling (`vector-db`, `rag`, `structured-extraction`, `prompt-engineering`) for the stated
+reason; then grade the `capability` scenario by checking the produced answer against every item
+in its `must_include` rubric. A pass covers all routing cases plus full rubric coverage on the
+capability scenario. The separate `scripts/verify.sh` lints produced embedding/eval artifacts
+and is independent of these cases.

package/skills/embeddings-search/evals/cases.yaml

@@ -0,0 +1,44 @@
+skill: embeddings-search
+
+should_trigger:
+  - prompt: "Which embedding model and chunk size should I use for these PDFs?"
+    why: "The core model + chunking decision this skill owns; no store or pipeline named."
+  - prompt: "My semantic search returns irrelevant results, how do I make retrieval better?"
+    why: "Relevance-quality symptom at the technique layer — model/chunking/hybrid/rerank, not store ops."
+  - prompt: "How do I measure if my retrieval is actually good — recall@k or nDCG?"
+    why: "Retrieval-quality measurement (golden set, recall/nDCG/MRR) is explicitly owned here."
+  - prompt: "Add a reranker and hybrid BM25+vector scoring to lift relevance."
+    why: "Hybrid fusion + reranker stage is a named section of this skill."
+  - prompt: "Exact-match queries like product SKUs return nothing but paraphrases work fine."
+    why: "Non-obvious: never names embeddings, but the diagnosis is a dense-only blind spot fixed by adding sparse/BM25 — squarely this skill."
+  - prompt: "recall@10 is fine but the top result is the wrong document, why?"
+    why: "Non-obvious ordering symptom — points at nDCG/MRR and a reranker, both owned here, not a store problem."
+  - prompt: "quin model d'embeddings i quina mida de chunk faig servir per a aquests documents legals"
+    why: "Catalan trigger on the model + chunk-size choice, the skill's primary decision."
+
+should_not_trigger:
+  - prompt: "Set up a Qdrant collection with HNSW and a metadata filter."
+    route_to: vector-db
+    why: "Collection schema and index/filter path are store operation, not embedding technique."
+  - prompt: "Build the full RAG pipeline: retrieve, prompt, and generate the grounded answer."
+    route_to: rag
+    why: "End-to-end answer loop plus groundedness eval is beyond where this skill stops (chunks returned)."
+  - prompt: "Pull the invoice number, date, and total out of these PDFs into JSON."
+    route_to: structured-extraction
+    why: "Typed field extraction from documents, not search/retrieval."
+  - prompt: "My Postgres <=> query is slow and returns fewer than k rows with a WHERE filter."
+    route_to: vector-db
+    why: "pgvector iterative-scan / index issue lives in the store layer."
+  - prompt: "Write a better system prompt so the model stops hallucinating tools."
+    route_to: prompt-engineering
+    why: "Prompt design, not retrieval quality."
+
+capability:
+  - scenario: "Semantic search over 50k multilingual support articles returns irrelevant results. The user doesn't know which embedding model or chunk size to use and has no way to tell whether a change helps."
+    must_include:
+      - "Picks a multilingual-capable model and justifies dims/cost, noting Matryoshka truncation as the lever instead of re-embedding"
+      - "Matches the distance metric to the model and embeds query vs document with the correct input_type asymmetry"
+      - "Gives a concrete chunking default (recursive ~512-token, token-accurate count, 10–20% overlap) and names when to upgrade (semantic/late/contextual) gated on metrics"
+      - "Adds hybrid dense+sparse fused by RRF (1/(k+rank), k≈60), then a reranker top-50→top-5, with the rule that a reranker cannot recover missed recall"
+      - "Builds a labeled golden query set and measures recall@k / nDCG@k (and MRR) before and after, changing exactly one variable"
+      - "Routes store/index setup to vector-db and the full answer/generation loop to rag"