@damiangilgonzalez/harness 1.3.7

package/skills/python/references/stdlib.md

@@ -0,0 +1,162 @@
+# Stdlib cookbook
+
+Read when reaching for the standard library before a dependency. Recipes for
+`pathlib`/`itertools`/`functools`/`collections`/`contextlib`, the `match` statement, the
+dataclass field machinery, and Enum patterns.
+
+## pathlib
+
+```python
+from pathlib import Path
+
+root = Path(__file__).resolve().parent
+cfg = root / "config" / "app.toml"        # join with /
+cfg.read_text(encoding="utf-8")
+cfg.with_suffix(".bak")                    # app.bak
+list(root.glob("**/*.py"))                 # recursive
+cfg.parent.mkdir(parents=True, exist_ok=True)
+```
+
+`Path` is typed, OS-independent, and composable — use it everywhere instead of `os.path` string
+joins. For just-a-string interop, `str(path)` or pass the `Path` directly (most stdlib accepts it).
+
+## collections
+
+```python
+from collections import defaultdict, Counter, deque
+
+groups: defaultdict[str, list[int]] = defaultdict(list)
+groups[key].append(n)                      # no KeyError, auto-creates the list
+
+counts = Counter(words)
+counts.most_common(3)                      # top 3 (word, count)
+
+window: deque[int] = deque(maxlen=100)     # fixed-size ring buffer; old items drop off
+window.append(x)
+```
+
+`ChainMap` layers several dicts (defaults <- env <- overrides) into one lookup.
+
+## itertools
+
+```python
+from itertools import chain, groupby, islice, batched, pairwise
+
+list(chain(a, b, c))                       # flatten iterables
+list(islice(gen, 10))                      # first 10 of a (possibly infinite) generator
+for size, group in groupby(rows, key=lambda r: r.size):   # input MUST be pre-sorted by key
+    ...
+list(batched(range(10), 3))                # 3.12+: [(0,1,2),(3,4,5),(6,7,8),(9,)]
+list(pairwise([1, 2, 3]))                  # [(1,2),(2,3)]
+```
+
+## functools
+
+```python
+from functools import cache, cached_property, partial, reduce
+
+@cache                                     # unbounded memo for a pure function
+def fib(n: int) -> int:
+    return n if n < 2 else fib(n - 1) + fib(n - 2)
+
+class Report:
+    @cached_property                       # computed once per instance, then stored
+    def total(self) -> int:
+        return sum(self.rows)
+
+add5 = partial(add, 5)                     # pre-bind an argument
+```
+
+Prefer `@cache` over `@lru_cache(maxsize=None)`; keep `@lru_cache(maxsize=N)` only when you need
+a bound.
+
+## contextlib
+
+```python
+from contextlib import contextmanager, ExitStack, suppress
+
+@contextmanager
+def timer(label: str):
+    start = time.perf_counter()
+    try:
+        yield
+    finally:
+        log.info("%s took %.3fs", label, time.perf_counter() - start)
+
+with ExitStack() as stack:                 # dynamic, variable number of resources
+    files = [stack.enter_context(open(p)) for p in paths]
+    # all closed on block exit, in reverse order
+
+with suppress(FileNotFoundError):          # ignore one specific error, cleanly
+    Path("cache").unlink()
+```
+
+## The match statement
+
+Structural pattern matching — dispatch on shape, not a chain of `isinstance`.
+
+```python
+def describe(value: object) -> str:
+    match value:
+        case []:
+            return "empty list"
+        case [x]:
+            return f"one item: {x}"
+        case [first, *rest]:
+            return f"{first} and {len(rest)} more"
+        case {"type": "point", "x": int(x), "y": int(y)}:
+            return f"point {x},{y}"
+        case Point(x=0, y=0):              # class pattern, captures by attribute
+            return "origin"
+        case str() | bytes():
+            return "text-ish"
+        case _:
+            return "other"
+```
+
+Capture patterns bind names; `case _` is the wildcard. Guard with `case x if x > 0:`.
+
+## dataclasses, in depth
+
+```python
+from dataclasses import dataclass, field
+
+@dataclass(frozen=True, slots=True, kw_only=True)
+class Order:
+    id: str
+    items: list[str] = field(default_factory=list)   # NEVER `= []` (shared mutable default)
+    total: float = 0.0
+    _cache: dict[str, int] = field(default_factory=dict, compare=False, repr=False)
+
+    def __post_init__(self) -> None:
+        if self.total < 0:                 # frozen: validate, don't mutate
+            raise ValueError("total < 0")
+```
+
+- `frozen=True` -> immutable + hashable. `slots=True` -> no `__dict__`, less memory, faster attrs.
+- `kw_only=True` -> all fields keyword-only (avoids positional-order foot-guns).
+- `field(default_factory=...)` for any mutable default; `compare=False`/`repr=False` to exclude a field.
+- `__post_init__` for validation/derived values (assign via `object.__setattr__` when frozen).
+
+## Enum patterns
+
+```python
+from enum import Enum, StrEnum, IntEnum, IntFlag, auto
+
+class Color(Enum):
+    RED = auto()
+    GREEN = auto()
+
+class Status(StrEnum):                     # 3.11+: members ARE strings
+    ACTIVE = "active"
+    CLOSED = "closed"
+# Status.ACTIVE == "active" -> True; great for JSON/DB string columns.
+
+class Perm(IntFlag):                       # bitwise-combinable flags
+    READ = auto()
+    WRITE = auto()
+    ALL = READ | WRITE
+```
+
+Use `StrEnum`/`IntEnum` when the value must interop as a primitive (JSON, DB); plain `Enum` for
+opaque tokens. Pair an `Enum` with `Literal` types at boundaries for exhaustiveness checking.

package/skills/python/references/typing.md

@@ -0,0 +1,160 @@
+# Typing — deep dive
+
+Read when the SKILL.md typing section is not enough: generic bounds/variance, structural vs
+nominal typing, `TypedDict` precision, callable-shape generics, narrowing, and the common
+`mypy --strict` errors and their fixes.
+
+## PEP 695 generics in full (Python 3.12+)
+
+Inline type parameters replace explicit `TypeVar` objects for new code.
+
+```python
+# Bound: T must be (a subtype of) Comparable.
+from typing import Protocol
+
+class Comparable(Protocol):
+    def __lt__(self, other: object) -> bool: ...
+
+def smallest[T: Comparable](xs: list[T]) -> T:
+    return min(xs)
+
+# Constraints: T is exactly one of these types (not "any subtype").
+def join[T: (str, bytes)](sep: T, parts: list[T]) -> T:
+    return sep.join(parts)
+
+# Generic class and a generic method on it.
+class Stack[T]:
+    def __init__(self) -> None:
+        self._items: list[T] = []
+    def push(self, item: T) -> None:
+        self._items.append(item)
+    def pop(self) -> T:
+        return self._items.pop()
+
+# The `type` statement: a true alias, lazily evaluated, can be generic.
+type Pair[T] = tuple[T, T]
+type Json = dict[str, "Json"] | list["Json"] | str | int | float | bool | None
+```
+
+**Variance** is inferred by the checker from how a type parameter is used — you no longer
+declare `covariant=`/`contravariant=`. A `list[T]` is invariant; a read-only `Sequence[T]`
+behaves covariantly. When you need an explicit infer-variance marker, PEP 695 exposes it, but
+in practice let the checker decide and fix the reported error.
+
+Legacy `TypeVar`/`Generic` syntax is **not deprecated** and may be mixed with the new syntax
+in the same file — but write PEP 695 for anything new.
+
+## Protocol (structural) vs ABC (nominal)
+
+- **`Protocol`**: duck typing the checker enforces. A class satisfies it by *shape*, with no
+  inheritance. Use for "anything with these methods". Add `@runtime_checkable` only if you need
+  `isinstance` against it (it checks method presence, not signatures).
+- **ABC** (`abc.ABC` + `@abstractmethod`): nominal — subclasses must explicitly inherit. Use
+  when you own the hierarchy and want to *share implementation* or force registration.
+
+```python
+from typing import Protocol, runtime_checkable
+
+@runtime_checkable
+class Closeable(Protocol):
+    def close(self) -> None: ...
+
+def shut(x: Closeable) -> None:
+    x.close()        # any object with close()->None type-checks; no base class needed
+```
+
+Prefer `Protocol` for boundaries you consume from elsewhere; reserve ABCs for frameworks you author.
+
+## TypedDict — Required / NotRequired
+
+Types a plain dict (e.g. a JSON payload) without a runtime class.
+
+```python
+from typing import TypedDict, Required, NotRequired
+
+class Config(TypedDict):
+    name: Required[str]
+    retries: NotRequired[int]     # key may be absent
+
+# total=False flips the default: every key NotRequired unless marked Required.
+class Patch(TypedDict, total=False):
+    name: str
+    retries: int
+```
+
+For closed value sets inside a dict field, combine with `Literal`. When you need *runtime
+validation* of the dict (untrusted input), that is a Pydantic/service concern -> `fastapi`.
+
+## Callable shapes: ParamSpec and TypeVarTuple
+
+```python
+from collections.abc import Callable
+from functools import wraps
+
+# ParamSpec: preserve the wrapped function's full signature through a decorator.
+def timed[**P, R](fn: Callable[P, R]) -> Callable[P, R]:
+    @wraps(fn)
+    def inner(*args: P.args, **kwargs: P.kwargs) -> R:
+        return fn(*args, **kwargs)
+    return inner
+
+# TypeVarTuple: variadic generics over an arbitrary number of types.
+def first_and_rest[T, *Ts](x: T, *rest: *Ts) -> tuple[T, tuple[*Ts]]:
+    return x, rest
+```
+
+## @overload — signature families
+
+When one function has distinct return types per argument shape, declare overloads then a single
+untyped-to-the-caller implementation:
+
+```python
+from typing import overload
+
+@overload
+def get(key: str) -> str: ...
+@overload
+def get(key: str, default: int) -> str | int: ...
+def get(key: str, default: int | None = None) -> str | int:
+    ...
+```
+
+## Narrowing: guards, TypeGuard, TypeIs
+
+Mypy follows control flow, so `isinstance`, `is None`, and `assert` narrow automatically.
+For custom predicates use `TypeIs` (3.13+, narrows in both branches) or `TypeGuard`:
+
+```python
+from typing import TypeIs
+
+def is_str_list(xs: list[object]) -> TypeIs[list[str]]:
+    return all(isinstance(x, str) for x in xs)
+
+def handle(xs: list[object]) -> None:
+    if is_str_list(xs):
+        xs[0].upper()        # narrowed to list[str] here
+```
+
+Use `typing.cast(T, value)` only as a last resort when you know more than the checker; it is an
+unchecked assertion. `reveal_type(x)` (no import needed under mypy) prints the inferred type at
+check time — delete it before committing.
+
+## Common `mypy --strict` errors and fixes
+
+| Error | Fix |
+| --- | --- |
+| `Function is missing a return type annotation` | Add `-> T` (use `-> None` for procedures). |
+| `Need type annotation for "x"` | Annotate the empty container: `x: list[int] = []`. |
+| `Returning Any from function declared to return "T"` | Type the source, or `cast` if truly unknowable. |
+| `Incompatible default for argument` | A `None` default needs `X | None` in the type. |
+| `Item "None" of "X | None" has no attribute ...` | Narrow with `if v is None: return` first. |
+| `Untyped decorator makes function "f" untyped` | Type the decorator with `ParamSpec` (see above). |
+| `Call to untyped function in typed context` | Add hints to the callee, or stub it. |
+
+## 3.14: deferred annotations
+
+Annotations are evaluated **lazily by default** (PEP 649/749): forward references resolve
+without `from __future__ import annotations`, recursive type aliases just work, and the new
+`annotationlib` module inspects annotations as live objects (`get_annotations`). `types.UnionType`
+is now an alias for `typing.Union`. Drop the `__future__` import in 3.14-only code; keep it for
+3.12/3.13 compatibility.

package/skills/python/scripts/verify.sh

@@ -0,0 +1,125 @@
+#!/usr/bin/env bash
+#
+# verify.sh - local quality gate for a Python project (single mirror of CI).
+#
+# Usage:
+#   cd <your-project-root>   # the directory containing pyproject.toml
+#   ./verify.sh
+#
+# Runs: uv sync --frozen, ruff check, ruff format --check, mypy --strict (or ty),
+# and pytest. Tools/dirs that are absent are skipped with a yellow warning, never a
+# failure. Real problems (lint, format drift, type errors, test failures) exit non-zero.
+# Read-only by default: it does not reformat or autofix your source.
+#
+# Portability: stock macOS bash 3.2 (no mapfile, no associative arrays, guarded under set -u).
+
+set -euo pipefail
+
+if [ -t 1 ]; then
+  YELLOW=$'\033[33m'; RED=$'\033[31m'; GREEN=$'\033[32m'; RESET=$'\033[0m'
+else
+  YELLOW=''; RED=''; GREEN=''; RESET=''
+fi
+
+failed=0
+
+have()  { command -v "$1" >/dev/null 2>&1; }
+warn()  { printf '%s[skip]%s %s\n' "$YELLOW" "$RESET" "$*"; }
+fail()  { printf '%s[fail]%s %s\n' "$RED" "$RESET" "$*"; failed=1; }
+ok()    { printf '%s[ ok ]%s %s\n' "$GREEN" "$RESET" "$*"; }
+info()  { printf -- '----- %s\n' "$*"; }
+
+# Must run from a project root.
+if [ ! -f pyproject.toml ]; then
+  printf '%serror:%s no pyproject.toml in %s - cd into your project root first.\n' \
+    "$RED" "$RESET" "$(pwd)" >&2
+  exit 2
+fi
+
+# Choose a runner: prefer `uv run <tool>` when uv is present, else the bare tool.
+have_uv=0
+if have uv; then have_uv=1; fi
+
+run_tool() {
+  # run_tool <toolname> [args...] : invoke via uv when available, else directly.
+  local tool="$1"; shift
+  if [ "$have_uv" -eq 1 ]; then
+    uv run "$tool" "$@"
+  else
+    "$tool" "$@"
+  fi
+}
+tool_present() {
+  # tool_present <toolname> : true if reachable directly or via uv.
+  if have "$1"; then return 0; fi
+  if [ "$have_uv" -eq 1 ] && uv run "$1" --version >/dev/null 2>&1; then return 0; fi
+  return 1
+}
+
+# 1. uv sync --frozen - reproducible install from the committed lockfile.
+info "uv sync"
+if [ "$have_uv" -eq 1 ]; then
+  if [ -f uv.lock ]; then
+    if uv sync --frozen; then ok "uv sync --frozen"; else fail "uv sync --frozen failed"; fi
+  else
+    warn "no uv.lock; trying uv sync (will create one)"
+    if uv sync; then ok "uv sync"; else fail "uv sync failed"; fi
+  fi
+else
+  warn "uv not found (https://docs.astral.sh/uv/); using whatever is on PATH"
+fi
+
+# 2. ruff check - lint. Read-only here (no --fix); CI should not mutate source.
+info "ruff check"
+if tool_present ruff; then
+  if run_tool ruff check .; then ok "ruff check clean"; else fail "ruff reported lint issues (run: ruff check --fix .)"; fi
+else
+  warn "ruff not found (uv add --dev ruff)"
+fi
+
+# 3. ruff format --check - formatting must be clean, but do not rewrite files.
+info "ruff format --check"
+if tool_present ruff; then
+  if run_tool ruff format --check .; then ok "ruff format clean"; else fail "formatting drift (run: ruff format .)"; fi
+else
+  warn "ruff not found - skipping format check"
+fi
+
+# 4. type check: mypy --strict, or ty if the project opted into it via [tool.ty].
+info "type check"
+type_target="."
+if [ -d src ]; then type_target="src"; fi
+if grep -q '^\[tool\.ty\]' pyproject.toml 2>/dev/null && tool_present ty; then
+  if run_tool ty check "$type_target"; then ok "ty clean"; else fail "ty reported type errors"; fi
+elif tool_present mypy; then
+  if run_tool mypy --strict "$type_target"; then ok "mypy --strict clean ($type_target)"; else fail "mypy --strict reported type errors"; fi
+else
+  warn "no type checker found (uv add --dev mypy)"
+fi
+
+# 5. pytest - only if there are tests to run.
+info "pytest"
+has_tests=0
+if [ -d tests ]; then has_tests=1; fi
+# Any test_*.py / *_test.py anywhere also counts.
+if [ "$has_tests" -eq 0 ]; then
+  if find . -type d -name .venv -prune -o -type f \( -name 'test_*.py' -o -name '*_test.py' \) -print 2>/dev/null | grep -q .; then
+    has_tests=1
+  fi
+fi
+if [ "$has_tests" -eq 1 ]; then
+  if tool_present pytest; then
+    if run_tool pytest -q; then ok "pytest passed"; else fail "pytest failed"; fi
+  else
+    warn "pytest not found (uv add --dev pytest)"
+  fi
+else
+  warn "no tests found - skipping pytest"
+fi
+
+echo
+if [ "$failed" -ne 0 ]; then
+  printf '%sFAIL:%s one or more checks failed.\n' "$RED" "$RESET"
+  exit 1
+fi
+printf '%sPASS:%s all checks passed.\n' "$GREEN" "$RESET"

package/skills/rag/SKILL.md

@@ -0,0 +1,209 @@
+---
+name: rag
+description: "Use when building grounded Q&A over your own corpus — chunk, retrieve hybrid, rerank, ground, cite chunk ids, refuse when the sources fall short — or when the right document is retrieved but the answer is still wrong, invented, or unmeasured. NOT operating the store itself — collection schema, HNSW ef_search, quantization (that is `vector-db`)."
+tags: [rag, retrieval-augmented-generation, chunking, hybrid-search, reranking, grounding, citations, faithfulness, contextual-retrieval]
+recommends: [vector-db, embeddings-search, document-processing, chatbot, agent-eval]
+origin: risco
+---
+
+# rag — own the retrieve → rerank → ground → cite → refuse pipeline
+
+You own the **pipeline** that turns a corpus plus a question into a grounded, cited answer: chunk,
+optionally contextualize, index, retrieve hybrid, rerank, assemble a grounded prompt, cite the
+sources, and **refuse** when the context does not contain the answer.
+
+You are judged by **retrieval quality and answer faithfulness**, not by raw vector math. If you
+find yourself tuning HNSW parameters, you wandered into the store underneath you
+(`../vector-db/SKILL.md`). If you are comparing embedding models or chunk sizes, that is the
+science beside you (`embeddings-search`).
+
+## The pipeline, and where each stage hands off
+
+Each stage is a real branch — most failures live in one specific stage, and several stages
+delegate to a sibling skill rather than living here.
+
+| Stage | What you do | Hands off to |
+|---|---|---|
+| Ingest | Get clean text out of PDFs/DOCX/HTML/OCR | you assume text exists → `../document-processing/SKILL.md` |
+| Chunk | Heading/semantic-aware splits with overlap, stable ids | model, dims + chunk-size science → `embeddings-search` |
+| Contextualize | Prepend an LLM-written context blurb per chunk (optional) | stays here |
+| Index | Embed + write dense vectors and a BM25/keyword index | you upsert, it owns the knobs → `../vector-db/SKILL.md` |
+| Retrieve | Hybrid dense + BM25, fuse with RRF, top ~150 | hybrid query mechanics → `../vector-db/SKILL.md` |
+| Rerank | Cross-encoder over the 150, keep top ~20 | stays here |
+| Ground + cite | System prompt: answer only from context, cite chunk ids | stays here |
+| Refuse | Output "I don't have enough information" on weak context | stays here |
+| Evaluate | Faithfulness, answer relevancy, context precision/recall | general harness → `agent-eval` |
+
+Three neighbors are not stages at all. Surfacing this answer inside a chat product (sessions,
+channels, UI) is `../chatbot/SKILL.md` — it calls you, not the reverse. A multi-step tool loop with
+state, where retrieval is one tool among many, is `../building-agents/SKILL.md`. Pulling
+schema-constrained fields out of text instead of a grounded prose answer is
+`structured-extraction`. `rag` is the retrieval brain those products call.
+
+## Retrieval is the bottleneck — measure recall before touching the prompt
+
+Naive RAG pipelines fail at the **retrieval** step in up to ~40% of cases *even when the correct
+document is in the corpus* (StackAI / Lushbinary, 2026-06-02). Why this matters: if the right
+passage never reaches the model, no prompt wording can save the answer. So your first move on a
+broken pipeline is never the prompt — it is measuring whether retrieval delivered the goods.
+
+```text
+Bad   → "answers are wrong, let me lower temperature and reword the system prompt."
+Good  → measure context recall on a golden set; if the right chunk isn't in the top-K,
+         fix chunking + hybrid + rerank first. Only then touch grounding.
+```
+
+Order of attack when answers are wrong: **context recall → context precision → grounding prompt
+→ generation params**. The last item almost never moves the needle.
+
+## Chunking — the highest-leverage single fix
+
+Chunk on structure, not on a blind character count. Why: too-small loses the context a passage
+needs to be interpretable; too-large dilutes the embedding so the relevant sentence gets averaged
+away (StackAI; EdenAI 2025, accessed 2026-06-02).
+
+- Split on headings/sections first, then sub-split long sections to a target window.
+- Keep **overlap** (~10–20% of the window) so a fact spanning a boundary survives in one chunk.
+- Attach a **stable `chunk_id` and `source`** at creation — you will need them end to end for
+  citations (see below). Never embed text and discard where it came from.
+
+```python
+# Heading-aware split sketch; real size tuning belongs in embeddings-search.
+def chunk_markdown(doc_id, text, target=800, overlap=120):
+    sections, buf, head = [], [], None
+    for line in text.splitlines():
+        if line.startswith("#"):
+            if buf: sections.append((head, "\n".join(buf))); buf = []
+            head = line.lstrip("# ").strip()
+        else:
+            buf.append(line)
+    if buf: sections.append((head, "\n".join(buf)))
+    out, i = [], 0
+    for head, body in sections:
+        for start in range(0, max(1, len(body)), target - overlap):
+            piece = body[start:start + target]
+            out.append({"chunk_id": f"{doc_id}#{i}", "source": doc_id,
+                        "heading": head, "text": piece})
+            i += 1
+    return out
+```
+
+The "what window/overlap maximizes recall for *this* corpus" study is `embeddings-search`; here you
+just need structurally sane chunks that keep their ids.
+
+## Contextual retrieval — prepend context before you embed
+
+Anthropic's Contextual Retrieval (Sept 2024) prepends a short LLM-generated blurb to each chunk
+*before* embedding **and** before BM25 indexing, so an isolated chunk knows what document and
+section it belongs to. Why it matters: it cuts failed retrievals by ~35% (contextual embeddings
+alone), ~49% (contextual embeddings + contextual BM25), and ~67% once reranking is added
+(anthropic.com/news/contextual-retrieval, 2024-09; accessed 2026-06-02).
+
+```text
+<document>{{WHOLE_DOC}}</document>
+Here is the chunk we want to situate within the whole document:
+<chunk>{{CHUNK}}</chunk>
+Give a short, standalone context (1–2 sentences) that situates this chunk within the
+document for search retrieval. Answer only with the context, nothing else.
+```
+
+Embed `context + "\n" + chunk_text` (not the bare chunk). The full implementation — caching the
+document prompt, batching, the BM25 side, and a runnable retrieve → rerank → answer skeleton —
+lives in `references/pipeline.md`.
+
+## Hybrid retrieval + RRF + rerank
+
+Dense vectors miss exact terms (codes, names, error strings); BM25 keyword search catches them
+but misses paraphrase. Combine both, fuse with **Reciprocal Rank Fusion (RRF)**, then rerank
+with a cross-encoder. The default-best quality/cost funnel (Microsoft Cloud Blog 2025-02-04;
+StackAI, accessed 2026-06-02):
+
+```text
+retrieve ~150 candidates (dense + BM25, fused with RRF)
+   → rerank all 150 with a cross-encoder
+   → keep top ~20 for the prompt
+```
+
+The actual hybrid query (named vectors, sparse-dense, server-side fusion) is `vector-db`. Here
+you own the funnel and the reranker choice:
+
+- **Cohere Rerank 3.5** — managed cross-encoder, context length 4096, SOTA on BEIR and
+  multilingual, available via Cohere API, Bedrock, Pinecone, Azure (docs.cohere.com/changelog/
+  rerank-v3.5, accessed 2026-06-02). Use when you want quality without hosting a model.
+- A **local cross-encoder** (e.g. a `bge-reranker`) — use when data cannot leave your network or
+  you need zero per-call cost; you pay in GPU/latency instead.
+
+```python
+# Rerank the fused candidates down to the prompt set; keep ids intact.
+import cohere
+co = cohere.ClientV2()
+ranked = co.rerank(model="rerank-v3.5", query=q,
+                   documents=[c["text"] for c in candidates], top_n=20)
+top = [candidates[r.index] for r in ranked.results]  # each still carries chunk_id + source
+```
+
+## Ground the prompt — answer only from context, cite, refuse
+
+Properly grounded RAG reduces hallucination rates by up to ~71%; poorly grounded pipelines still
+hallucinate in up to ~40% of responses *even with the right doc retrieved* (Confident AI / Maxim
+2025, accessed 2026-06-02). The prompt must do three things: bind the answer to the context, force
+inline citations, and provide an explicit refusal path.
+
+```text
+You answer ONLY using the information inside <context>. Do not use prior knowledge.
+Cite every claim with the chunk id it came from, like [chunk_id]. Multiple ids are fine.
+If the context does not contain enough information to answer, reply exactly:
+"I don't have enough information in the provided sources to answer that."
+(Spanish corpora: "No tengo suficiente información en las fuentes para responder.")
+
+<context>
+[doc12#3] {chunk text...}
+[doc12#4] {chunk text...}
+</context>
+
+Question: {{question}}
+```
+
+A grounding prompt **without a refusal path is a bug** — it converts "missing context" into a
+confident fabrication. The refusal clause is what turns retrieval failures into honest non-answers.
+
+## Citations — carry ids the whole way
+
+The answer can only link back if a stable id survives every stage: chunk → retrieve → rerank →
+prompt → answer. Why: if you embed text and drop the id at index time, there is nothing for a
+citation to point at, and you cannot debug which passage produced a wrong claim.
+
+- Put `chunk_id` and `source` on the chunk at creation; keep them on the object through rerank.
+- Render them into the `<context>` block (`[chunk_id] text`) so the model can quote them.
+- Map cited ids back to source URLs/pages when you render the answer to the user.
+
+## Evaluating it — metrics, not vibes
+
+Faithfulness is **not** correctness: an answer can be faithful to a wrong chunk. Measure four
+RAGAS metrics on a small golden Q/A set (RAGAS docs; Cohorte 2025; Confident AI, accessed
+2026-06-02):
+
+| Failure symptom | Metric that catches it | First fix |
+|---|---|---|
+| Answer states things not in the sources | **faithfulness** (claims supported by context) | grounding prompt + refusal |
+| Answer is on-topic but doesn't address the question | **answer relevancy** | prompt / query rewriting |
+| Relevant chunks exist but rank below junk | **context precision** | reranker, RRF weights |
+| The needed chunk never gets retrieved | **context recall** | chunking, contextual retrieval, hybrid |
+
+Build a 30–50 question golden set with known-good answers, score with RAGAS, and gate CI below a
+threshold (e.g. faithfulness ≥ 0.90, context recall ≥ 0.85). Full formulas, thresholds, and the
+CI snippet are in `references/evaluation.md`. The general-purpose eval harness is `agent-eval`;
+the RAG-specific metrics live here.
+
+## Anti-patterns
+
+| Anti-pattern | Why it breaks | Do instead |
+|---|---|---|
+| Fixed char-count chunking, blind to structure | Splits mid-sentence; dilutes embeddings | Heading/semantic chunks with overlap |
+| Rerank disabled — stuff top-50 raw into prompt | Noise drowns the right passage; cost balloons | Retrieve ~150 → rerank → keep ~20 |
+| No refusal path in the grounding prompt | Missing context becomes confident fabrication | Explicit "I don't have enough information" |
+| Embed text, drop the chunk id | Nothing to cite or debug | Carry `chunk_id`+`source` end to end |
+| "It looks good" eval on vibes | Regressions ship silently | Golden set + RAGAS + CI threshold gate |
+| Embedding the query differently from the corpus | Query and chunks land in different spaces | Same model + same preprocessing both sides |
+| Dense-only, ignoring BM25/keyword | Misses exact codes/names/error strings | Hybrid dense + BM25 fused with RRF |
+| Tuning temperature to fix wrong answers | Generation is rarely the bottleneck | Measure context recall first |

package/skills/rag/evals/README.md

@@ -0,0 +1,13 @@
+# rag evals — how to run
+
+These are routing and capability checks, not unit tests. Run the `should_trigger` prompts against
+your skill router and confirm `rag` fires; run the `should_not_trigger` prompts and confirm the
+router instead defers to the named sibling (`vector-db`, `embeddings-search`, `document-processing`,
+`chatbot`, `structured-extraction`) — a miss there means the boundary in the description needs
+sharpening. For the `capability` case, hand the scenario to the agent with this skill loaded and
+grade the answer against the `must_include` rubric line by line (does it name all five pipeline
+stages, hybrid + RRF, the rerank funnel, the grounding-plus-refusal contract, the eval gate, and
+the two handoffs); each rubric item is pass/fail. You can do this by hand or wire it through the
+`agent-eval` harness for a repeatable score. `scripts/verify.sh` is the static gate on the skill's
+own artifacts (frontmatter, the grounding contract, the references, the RAGAS metric names) and is
+separate from these capability checks.