@dsalazardev/ai-hub-cli 0.0.0-stage → 0.1.0

package/skills/shared/agent-eval/references/judge-design.md

@@ -0,0 +1,118 @@
+# LLM-as-judge design and calibration
+
+Depth offloaded from `SKILL.md`. A judge is a scorer, so the one rule binds hardest here:
+**calibrate against human gold and report agreement before you trust a single judge score.**
+Judge–human agreement reaches ~85% in 2026 (higher than two humans on the same task) — but
+only with a capable judge model and a rubric that forces a written rationale.
+
+## Pointwise rubric template
+
+Pointwise (rate one answer) is convenient but inflates and clusters at 8–9. Use it only when
+you cannot run pairwise, and always force the rationale first.
+
+```text
+You are grading an answer against a reference. The dimension is FAITHFULNESS:
+every claim in the answer must be supported by the provided context.
+
+Context:
+{context}
+
+Answer:
+{answer}
+
+Steps (do them in order):
+1. List each factual claim in the answer.
+2. For each claim, mark SUPPORTED or UNSUPPORTED against the context.
+3. Only then output a JSON object: {"rationale": "...", "score": <0.0-1.0>}
+   where score = supported_claims / total_claims.
+```
+
+The order matters: a rubric that asks for the number first gets a vibe; asking for the
+claim-by-claim breakdown first forces the reasoning that earns the ~85% agreement.
+
+## Pairwise rubric template (preferred)
+
+"Is A or B better?" is more reproducible than an absolute rating. Use it for before/after and
+A/B comparisons.
+
+```text
+Compare two answers, A and B, against the reference for HELPFULNESS.
+
+Reference: {reference}
+Answer A: {a}
+Answer B: {b}
+
+1. One sentence on what A does well/badly vs the reference.
+2. One sentence on what B does well/badly vs the reference.
+3. Output JSON: {"rationale": "...", "winner": "A" | "B" | "tie"}
+```
+
+## Position-swap harness (kills position bias)
+
+Judges favor whichever answer appears first. Run each comparison twice with positions swapped
+and only count a clear win as a win.
+
+```python
+def pairwise(judge, reference, a, b):
+    """Return 'a', 'b', or 'tie' after canceling position bias."""
+    first  = judge.compare(reference, a, b)   # A in slot 1
+    second = judge.compare(reference, b, a)   # A in slot 2 -> remap
+    remap = {"A": "b", "B": "a", "tie": "tie"}
+    r1 = {"A": "a", "B": "b", "tie": "tie"}[first]
+    r2 = remap[second]
+    if r1 == r2:
+        return r1                # consistent across positions -> trust it
+    return "tie"                 # judge flipped with position -> not a real signal
+```
+
+A judge that flips when you swap positions has told you it cannot separate the two — record a
+tie, do not pick the first-slot winner.
+
+## Calibration script (agreement and Cohen's kappa vs human gold)
+
+You need a small human-labeled gold slice (~10% of cases, the ambiguous ones). Run the judge
+on it and measure how often it agrees with the humans. Report this number in `eval-report.json`
+and refuse to gate if it is low.
+
+```python
+def cohen_kappa(human: list[int], judge: list[int]) -> float:
+    """Agreement corrected for chance. 1.0 perfect, 0 chance-level."""
+    n = len(human)
+    po = sum(h == j for h, j in zip(human, judge)) / n        # raw agreement
+    labels = set(human) | set(judge)
+    pe = sum((human.count(k) / n) * (judge.count(k) / n) for k in labels)
+    return (po - pe) / (1 - pe) if pe != 1 else 1.0
+
+def calibration_report(human, judge):
+    raw = sum(h == j for h, j in zip(human, judge)) / len(human)
+    return {"raw_agreement": round(raw, 3),
+            "cohen_kappa": round(cohen_kappa(human, judge), 3)}
+```
+
+Rules of thumb for the kappa: < 0.4 the judge is unusable, fix the rubric or model; 0.4–0.6
+marginal, widen the human slice; > 0.6 with raw agreement ~0.85 is the working zone. These are
+thresholds for *trusting the judge*, not for gating the system.
+
+## G-Eval vs DAG
+
+- **G-Eval** — you give the judge an evaluation-criteria sentence; it generates the
+  chain-of-thought steps and a weighted score. Fast to author, good for fuzzy semantic
+  dimensions (coherence, helpfulness). Less reproducible for hard rules.
+- **DAG (Decision Graph)** — you author an explicit decision tree of yes/no checks; the score
+  is deterministic given the answers. Use for compliance-style "must / must-not" criteria where
+  you want auditability over flexibility. DeepEval v4 ships this as first-class.
+
+Pick G-Eval for taste, DAG for rules.
+
+## Judge bias catalog and mitigations
+
+| Bias | Symptom | Mitigation |
+| --- | --- | --- |
+| Position | Favors the first answer shown | Position-swap harness above; count only consistent wins |
+| Length | Scores longer answers higher regardless of quality | Add "ignore length; reward density" to the rubric; spot-check long losers |
+| Self-preference | A model prefers text in its own style | Use a different model family as judge than the system under test |
+| Verbosity-of-rationale | Long rationale read as more correct | Score the claim breakdown, not the prose |
+| Anchoring on reference wording | Penalizes correct paraphrases | Grade meaning vs reference, not surface overlap; test with known-good paraphrases |
+
+Re-run the calibration slice whenever you change the judge model or rubric — a judge swap is a
+scorer change, and an uncalibrated scorer is back to a number you cannot defend.

package/skills/shared/agent-eval/references/runner-and-gate.md

@@ -0,0 +1,183 @@
+# The runner, the gate, and framework equivalents
+
+Depth offloaded from `SKILL.md`. A provider-neutral runner you can drop into any repo, the CI
+wiring, and the same eval expressed in DeepEval and Inspect AI so you can switch tools without
+relearning the craft.
+
+## The five stages, in code
+
+`dataset → runner → scorers → metrics → gate`. Keep them separable so each can be swapped.
+
+### Dataset loader (JSONL, decontaminated, versioned)
+
+```python
+import json
+from pathlib import Path
+
+REQUIRED = {"id", "input", "expected"}
+
+def load_cases(path: str) -> list[dict]:
+    cases = []
+    for i, line in enumerate(Path(path).read_text().splitlines(), 1):
+        line = line.strip()
+        if not line:
+            continue
+        case = json.loads(line)
+        missing = REQUIRED - case.keys()
+        if missing:
+            raise ValueError(f"{path}:{i} missing keys {missing}")
+        cases.append(case)
+    return cases
+```
+
+### Scorer registry
+
+Every scorer satisfies one protocol — deterministic and judge are interchangeable. See
+`judge-design.md` for the judge implementations.
+
+```python
+from dataclasses import dataclass
+from typing import Callable
+
+@dataclass
+class Scorer:
+    name: str
+    fn: Callable[[dict, str], float]   # (case, output) -> 0.0..1.0
+
+def exact_match(case, output):
+    return 1.0 if output.strip() == case["expected"].strip() else 0.0
+
+def latency_under(threshold_s):
+    def _fn(case, output):
+        return 1.0 if case["meta"].get("latency_s", 0) <= threshold_s else 0.0
+    return _fn
+
+REGISTRY = [
+    Scorer("exact_match", exact_match),
+    Scorer("latency_2s", latency_under(2.0)),
+    # Scorer("faithfulness", FaithfulnessJudge(judge_model).score),  # 30% judge
+]
+```
+
+### Runner + bootstrap-CI metrics
+
+The bootstrap CI is what stops judge noise from flapping the gate: only a drop beyond the
+interval counts as a real regression.
+
+```python
+import random, statistics
+
+def run(system, cases, scorers):
+    rows = []
+    for case in cases:
+        output = system(case["input"])               # the system under test
+        row = {"id": case["id"],
+               "failure_mode": case["meta"].get("failure_mode", "none")}
+        for s in scorers:
+            row[s.name] = s.fn(case, output)
+        rows.append(row)
+    return rows
+
+def metrics(rows, scorers, n_boot=1000):
+    out = {}
+    for s in scorers:
+        vals = [r[s.name] for r in rows]
+        mean = statistics.fmean(vals)
+        boots = [statistics.fmean(random.choices(vals, k=len(vals)))
+                 for _ in range(n_boot)]
+        boots.sort()
+        out[s.name] = {"mean": round(mean, 4),
+                       "ci_low": round(boots[int(0.025 * n_boot)], 4),
+                       "ci_high": round(boots[int(0.975 * n_boot)], 4)}
+    return out
+```
+
+### Gate (block on regression vs committed baseline)
+
+```python
+import json, sys
+
+def gate(current: dict, baseline: dict, report_path="eval-report.json") -> int:
+    regressed = []
+    for metric, m in current.items():
+        base = baseline.get(metric, {}).get("mean")
+        # Real regression: the current CI is entirely below the baseline mean.
+        if base is not None and m["ci_high"] < base:
+            regressed.append({"metric": metric, "baseline": base,
+                              "now": m["mean"], "ci_high": m["ci_high"]})
+    report = {"metrics": current, "baseline": baseline,
+              "regressed": regressed, "passed": not regressed}
+    Path(report_path).write_text(json.dumps(report, indent=2))
+    for r in regressed:
+        print(f"REGRESSION {r['metric']}: {r['baseline']} -> {r['now']}",
+              file=sys.stderr)
+    return 1 if regressed else 0
+```
+
+Update the baseline deliberately — commit a new `eval-baseline.json` in the PR that *intends*
+to move the metric, so the move is reviewed, not silent.
+
+## GitHub Actions wiring
+
+```yaml
+name: eval-gate
+on: [pull_request]
+jobs:
+  eval:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with: { python-version: "3.12" }
+      - run: pip install -r evals/requirements.txt
+      - name: Run eval gate
+        env:
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+        run: python evals/run.py          # exits non-zero on regression -> blocks merge
+      - uses: actions/upload-artifact@v4
+        if: always()
+        with: { name: eval-report, path: eval-report.json }
+```
+
+The non-zero exit is the gate. Upload the report `if: always()` so a failed run still shows
+*which* metric and failure mode regressed.
+
+## Same eval in DeepEval (pytest-native)
+
+```python
+from deepeval import assert_test
+from deepeval.test_case import LLMTestCase
+from deepeval.metrics import FaithfulnessMetric
+
+def test_faithfulness():
+    case = LLMTestCase(
+        input="Where is my refund for order 4821?",
+        actual_output=system("Where is my refund for order 4821?"),
+        retrieval_context=["policy: refunds 5-7 business days"],
+    )
+    assert_test(case, [FaithfulnessMetric(threshold=0.8)])
+```
+
+DeepEval v4.0.3 reads like tests and runs under pytest; reach for it when CI is already Python.
+
+## Same eval in Inspect AI (Task / Solver / Scorer)
+
+```python
+from inspect_ai import Task, task
+from inspect_ai.dataset import json_dataset
+from inspect_ai.solver import generate
+from inspect_ai.scorer import model_graded_qa
+
+@task
+def refund_quality():
+    return Task(
+        dataset=json_dataset("evals/cases.jsonl"),
+        solver=generate(),
+        scorer=model_graded_qa(),   # judge with rationale; bootstrap CIs built in
+    )
+# inspect eval refund_quality.py --model openai/gpt-... -> pass/fail + CIs
+```
+
+Inspect AI v0.3.225 (UK AISI) gives first-class tool-use and trajectory logging plus bootstrap
+CIs out of the box; reach for it when you are multi-provider or need real trajectory scoring.
+Both express the identical `dataset → scorer → gate` pipeline — the tool is a detail.

package/skills/shared/agent-eval/scripts/verify.sh

@@ -0,0 +1,161 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# verify.sh — agent-eval skill gate. Run from your PROJECT root.
+#
+# What it does (read-only, idempotent, NEVER calls an LLM or network):
+#   1. Discovers *.jsonl golden sets (skips vendor dirs); checks every line parses
+#      and carries the required keys id, input, expected. Hard fail on a bad line.
+#   2. If an eval-report.json exists, validates its shape (a "passed" boolean and a
+#      "metrics" object). Hard fail only on malformed JSON / missing shape.
+#   3. Optional lint, advisory only: ruff + mypy on example Python, markdownlint on docs.
+#
+# Exit non-zero ONLY on a malformed JSONL line, a missing required key, or a malformed
+# eval-report.json. Every optional lint and every missing tool is a yellow WARN/skip.
+# An empty or clean target exits 0. Stock macOS bash 3.2 (no mapfile / associative arrays).
+
+YELLOW=$'\033[33m'; GREEN=$'\033[32m'; RED=$'\033[31m'; NC=$'\033[0m'
+EXIT=0
+skip() { printf '%s[skip]%s %s\n' "$YELLOW" "$NC" "$*"; }
+note() { printf '%s[warn]%s %s\n' "$YELLOW" "$NC" "$*"; }
+ok()   { printf '%s[ok]%s %s\n'   "$GREEN"  "$NC" "$*"; }
+err()  { printf '%s[fail]%s %s\n' "$RED"    "$NC" "$*"; EXIT=1; }
+
+ROOT="$(pwd)"
+
+# Need a JSON-aware tool for the structural checks. Prefer python3, fall back to jq.
+JSON_TOOL=""
+if command -v python3 >/dev/null 2>&1; then JSON_TOOL="python3"
+elif command -v jq >/dev/null 2>&1; then JSON_TOOL="jq"; fi
+
+# --- 1. golden-set JSONL validation -----------------------------------------
+JSONL_FILES=()
+while IFS= read -r -d '' f; do
+  JSONL_FILES+=("$f")
+done < <(
+  find "$ROOT" \
+    \( -path '*/node_modules/*' -o -path '*/.git/*' -o -path '*/vendor/*' \
+       -o -path '*/.venv/*' -o -path '*/dist/*' \) -prune -o \
+    -type f -name '*.jsonl' -print0 2>/dev/null
+)
+
+if [ "${#JSONL_FILES[@]}" -eq 0 ]; then
+  skip "no *.jsonl golden sets found under $ROOT — nothing to validate"
+elif [ -z "$JSON_TOOL" ]; then
+  skip "neither python3 nor jq found — cannot validate JSONL, skipping"
+else
+  for f in "${JSONL_FILES[@]}"; do
+    if [ "$JSON_TOOL" = "python3" ]; then
+      msg="$(python3 - "$f" <<'PY'
+import json, sys
+path = sys.argv[1]
+req = {"id", "input", "expected"}
+bad = []
+with open(path, encoding="utf-8") as fh:
+    for n, line in enumerate(fh, 1):
+        s = line.strip()
+        if not s:
+            continue
+        try:
+            obj = json.loads(s)
+        except ValueError as e:
+            bad.append(f"line {n}: invalid JSON ({e})"); continue
+        if not isinstance(obj, dict):
+            bad.append(f"line {n}: not a JSON object"); continue
+        missing = req - obj.keys()
+        if missing:
+            bad.append(f"line {n}: missing keys {sorted(missing)}")
+print("\n".join(bad), end="")
+PY
+)" || msg="parser crashed on $f"
+      if [ -n "$msg" ]; then
+        err "$f:"; printf '  %s\n' "$msg"
+      else
+        ok "$f valid (id/input/expected present on every line)"
+      fi
+    else
+      # jq path: each line must be an object with the three keys.
+      if jq -e 'has("id") and has("input") and has("expected")' "$f" >/dev/null 2>&1; then
+        ok "$f valid (jq: required keys present)"
+      else
+        err "$f: a line is not an object or is missing id/input/expected (jq)"
+      fi
+    fi
+  done
+fi
+
+# --- 2. eval-report.json shape ------------------------------------------------
+REPORTS=()
+while IFS= read -r -d '' f; do
+  REPORTS+=("$f")
+done < <(
+  find "$ROOT" \
+    \( -path '*/node_modules/*' -o -path '*/.git/*' -o -path '*/.venv/*' \) -prune -o \
+    -type f -name 'eval-report.json' -print0 2>/dev/null
+)
+
+if [ "${#REPORTS[@]}" -eq 0 ]; then
+  skip "no eval-report.json found — gate output not validated"
+elif [ -z "$JSON_TOOL" ]; then
+  skip "no python3/jq — cannot validate eval-report.json shape"
+else
+  for f in "${REPORTS[@]}"; do
+    if [ "$JSON_TOOL" = "python3" ]; then
+      if python3 - "$f" <<'PY'
+import json, sys
+try:
+    r = json.load(open(sys.argv[1], encoding="utf-8"))
+except ValueError as e:
+    print(e); sys.exit(1)
+ok = isinstance(r, dict) and isinstance(r.get("passed"), bool) \
+     and isinstance(r.get("metrics"), dict)
+sys.exit(0 if ok else 1)
+PY
+      then ok "$f shape ok (passed: bool, metrics: object)"
+      else err "$f: malformed or missing 'passed'(bool)/'metrics'(object)"; fi
+    else
+      if jq -e 'type=="object" and (.passed|type=="boolean") and (.metrics|type=="object")' "$f" >/dev/null 2>&1; then
+        ok "$f shape ok (jq)"
+      else err "$f: malformed or missing passed/metrics (jq)"; fi
+    fi
+  done
+fi
+
+# --- 3. optional lint (advisory only) ----------------------------------------
+PY_FILES=()
+while IFS= read -r -d '' f; do
+  PY_FILES+=("$f")
+done < <(
+  find "$ROOT" \
+    \( -path '*/node_modules/*' -o -path '*/.git/*' -o -path '*/.venv/*' \) -prune -o \
+    -type f -name '*.py' -print0 2>/dev/null
+)
+
+if [ "${#PY_FILES[@]}" -gt 0 ]; then
+  if command -v ruff >/dev/null 2>&1; then
+    if ruff check "${PY_FILES[@]}" >/dev/null 2>&1; then ok "ruff clean"; else note "ruff reported lint findings (advisory)"; fi
+  else skip "ruff not installed — skipping Python lint"; fi
+  if command -v mypy >/dev/null 2>&1; then
+    mypy "${PY_FILES[@]}" >/dev/null 2>&1 && ok "mypy clean" || note "mypy reported type findings (advisory)"
+  else skip "mypy not installed — skipping type check"; fi
+else
+  skip "no *.py files — skipping ruff/mypy"
+fi
+
+MD_FILES=()
+while IFS= read -r -d '' f; do
+  MD_FILES+=("$f")
+done < <(
+  find "$ROOT" \
+    \( -path '*/node_modules/*' -o -path '*/.git/*' \) -prune -o \
+    -type f -name '*.md' -print0 2>/dev/null
+)
+if [ "${#MD_FILES[@]}" -gt 0 ] && command -v markdownlint >/dev/null 2>&1; then
+  markdownlint "${MD_FILES[@]}" >/dev/null 2>&1 && ok "markdownlint clean" || note "markdownlint findings (advisory)"
+else
+  skip "markdownlint not installed or no docs — skipping doc lint"
+fi
+
+printf '\n'
+if [ "$EXIT" -eq 0 ]; then ok "verify.sh passed"; else err "verify.sh found failures"; fi
+exit "$EXIT"

package/skills/shared/agent-safety/SKILL.md

@@ -0,0 +1,169 @@
+---
+name: agent-safety
+description: "Use when bounding an LLM agent that already runs — scoping its task domain, gating tools to least privilege, defending against prompt injection in untrusted web/email/RAG text, requiring human approval on irreversible actions, capping runtime and cost, or triaging what it already did. NOT building the loop, tools, or RAG (that is `building-agents`)."
+tags: [agent-security, guardrails, prompt-injection, least-privilege, owasp-agentic]
+recommends: [building-agents, secure-coding, agent-eval]
+origin: ai-hub
+---
+
+# Agent safety
+
+You are the security review for an agent's **agency**, not for its code. The loop works,
+tools are wired, memory persists — your job is to make that autonomy *bounded*. If you
+want to review ordinary endpoints, auth, or secrets handling, that is
+`../secure-coding/SKILL.md` — this skill is the *Agentic* Top 10, the risks that exist only
+because a model has tools and autonomy. If the loop or tools do not exist yet, that is
+`../building-agents/SKILL.md`. You arrive *after* both.
+
+`references/threat-model.md` carries the OWASP Agentic Top 10 2026 risks mapped to the
+controls below, the pre-ship guardrail checklist, and the incident-triage flow for "the
+agent did X" — open it when you are reviewing before ship or reconstructing an incident.
+
+## The ownership split
+
+Agent security splits into four layers — **Model · Harness · Tools · Environment**. The
+model provider owns only the Model layer (alignment, refusals). Everything else is yours:
+the Harness (loop, memory, context assembly), the Tools (what the agent can *do*), and the
+Environment (creds, network, blast radius). Do not outsource a layer you own to "the model
+is aligned."
+
+Three excesses cause almost every agentic incident. Cut all three:
+
+- **Excessive functionality** — tools the task never needs.
+- **Excessive permissions** — broader scopes/creds than the tool needs.
+- **Excessive autonomy** — acting without checking back when it should.
+
+The operating principle is **least agency**: autonomy is earned per task, not defaulted.
+
+## Scope limits
+
+- **Declare the allowed task domain as a hard boundary in the system prompt.** Why: an
+  undeclared scope is an infinite scope; "you are a refund assistant; you do not touch
+  payroll" is a constraint a reviewer can check.
+- **Deny by default — the agent starts with zero tools.** Each tool earns its place by a
+  task justification. Why: an opt-out tool list grows; an opt-in list stays minimal.
+- **Segregate the instruction channel from the data channel.** System/developer prompt =
+  trusted instructions. Everything the agent reads at runtime = data, never instructions.
+  Why: this single boundary is what stops indirect injection (LLM01).
+
+## Tool gating / least agency
+
+Give every tool a profile: **read / write / exec / send**, the exact resources it may
+touch, and an **allowlist** (never a wildcard). Block destructive flags and secret paths
+at the tool boundary, not in the prompt — the prompt is advisory, the boundary is enforced.
+
+```python
+# Bad: one wildcard tool = unbounded blast radius, runs anything the loop emits
+def run_shell(cmd: str) -> str:
+    return subprocess.run(cmd, shell=True, capture_output=True, text=True).stdout
+```
+
+```python
+# Good: narrow tool, allowlisted root, denied patterns, no shell
+ALLOWED_ROOT = pathlib.Path("/srv/agent/workspace").resolve()
+DENY = ("*.key", "*.pem", "*secret*", "*.env", "id_rsa*")
+
+def read_file(path: str) -> str:
+    p = (ALLOWED_ROOT / path).resolve()
+    if not p.is_relative_to(ALLOWED_ROOT):           # no traversal out of scope
+        raise PermissionError("path outside workspace")
+    if any(p.match(g) for g in DENY):                # never read secrets
+        raise PermissionError("denied pattern")
+    return p.read_text()
+```
+
+- **Issue task-scoped, short-lived tokens — not the session's broad creds.** A credential
+  should be valid only for the specific tool and the duration of one task. Why: a hijacked
+  loop cannot reuse a session-wide token it never held.
+- **Prefer read-only by default; writes/sends/exec are separate, gated tools.** Why: most
+  steps only need to read, so most steps should be unable to mutate anything.
+
+## Injection defense
+
+Treat **all** external data as untrusted: user messages, retrieved documents, API
+responses, emails, web pages, other agents' output. Sanitize and delimit before it enters
+context, and **never let external text reach a privileged tool unmediated.**
+
+| Source                         | Trust level | Required mediation before it can act        |
+| ------------------------------ | ----------- | ------------------------------------------- |
+| System / developer prompt      | Trusted     | none (this is the only instruction channel) |
+| End-user chat message          | Untrusted   | delimit; treat as data, not commands        |
+| Retrieved RAG / KB document    | Untrusted   | delimit; strip instruction-like spans       |
+| Fetched web page / API JSON    | Untrusted   | parse to schema; no raw text → tool args    |
+| Inbound email / ticket body    | Untrusted   | delimit; HITL on any action it requests     |
+| Another agent's message        | Untrusted   | same as external user input                 |
+
+```python
+# Bad: retrieved chunk flows straight into a privileged action
+chunk = retriever.search(q)[0].text          # attacker-controlled doc
+agent.call_tool("send_email", to=extract_to(chunk), body=chunk)
+```
+
+```python
+# Good: external content is quarantined data; the action is schema-validated + gated
+chunk = retriever.search(q)[0].text
+ctx = f"<retrieved untrusted>\n{chunk}\n</retrieved untrusted>"   # delimited, labeled
+proposal = agent.draft("send_email", context=ctx)                # model proposes
+args = SendEmail.model_validate(proposal.args)                   # schema or reject
+if args.to_domain not in ALLOWED_DOMAINS:                        # exfil guard
+    raise PermissionError("recipient outside allowlist")
+require_human_approval("send_email", args)                       # irreversible → HITL
+```
+
+- **Validate every tool-call argument against a strict schema before execution.** Why: a
+  schema rejects the surprise field, encoded payload, or off-allowlist recipient injection
+  produces.
+- **Watch for exfiltration shapes** — unexpected outbound URLs, base64 blobs, recipients
+  outside the allowlist. Why: data theft is the common payload of a successful injection.
+
+## Human-in-the-loop by risk class
+
+Do **not** approve every action — reported ~93% of permission prompts get approved without
+being read, so blanket prompting trains a rubber stamp. Gate by **risk class**, keyed on
+reversibility × blast radius. Bind each approval to the **exact parameters** with a
+short-lived token so the approved action cannot be swapped after the click.
+
+| Action type (examples)                          | Reversible? | Blast radius | Control            |
+| ----------------------------------------------- | ----------- | ------------ | ------------------ |
+| Read file, search KB, fetch page                | n/a         | none         | **auto**           |
+| Write to scratch workspace, internal draft      | yes         | local        | **log-only**       |
+| Mutate prod DB, deploy, change config            | hard        | system       | **approve (HITL)** |
+| Send email/payment to external party, post live | no          | external     | **approve (HITL)** |
+| Delete backups, rotate prod creds, mass-delete  | no          | catastrophic | **block** (or step-up auth) |
+
+- **Step up for the top row** — high-value irreversible actions deserve fresh auth, not the
+  ambient session. Why: a hijacked session should not also hold the keys to the worst action.
+
+## Memory hygiene
+
+- **Validate and sanitize content before it is stored.** Why: memory poisoning persists
+  across sessions (OWASP Agentic T1) — unlike session-scoped injection, a poisoned memory
+  re-attacks every future run until purged.
+- **Isolate memory per user and per session; do not let one user's writes color another's
+  reads.** Why: shared memory is a cross-tenant injection channel.
+- **Expire entries and cap memory size; redact PII (SSN, cards, API keys) before persist.**
+  Why: stale instructions and leaked secrets both age into liabilities.
+
+## Runtime kill-switches
+
+A looping or hijacked agent must hit a wall on its own. Set hard caps, fail closed:
+
+- **Tool-call rate cap** (e.g. ~30 calls/min) — runaway loops trip it before they do damage.
+- **Cost cap per session** (e.g. ~$10) — a wallet attack stops at a known ceiling.
+- **Loop / step cap** — a fixed max iterations kills the infinite plan.
+- **Wall-clock timeout** — a stuck agent is terminated, not left running.
+
+Log every tool call (arguments redacted) and alert on repeated approval-bypass attempts.
+
+## Anti-patterns
+
+| Anti-pattern                                   | Why it bites                                                   | Do instead                                          |
+| ---------------------------------------------- | ------------------------------------------------------------- | --------------------------------------------------- |
+| Approve every action                           | ~93% rubber-stamped; the real risky one slips through         | Gate by risk class; HITL only on irreversible/external |
+| One broad session token shared by all tools    | Hijacked loop reuses it everywhere                            | Task-scoped, short-lived per-tool tokens            |
+| Trust RAG / fetched / email content            | Indirect injection (LLM01) becomes direct tool execution      | Delimit as untrusted data; mediate before any tool  |
+| Wildcard `run_shell(cmd)` tool                 | Unbounded blast radius                                        | Narrow tools, allowlisted resources, denied patterns |
+| Raw external text piped into tool args         | Attacker controls the action's parameters                     | Schema-validate args; allowlist recipients/domains  |
+| Scope/limits stated only in the prompt         | Prompt is advisory; the model can be talked out of it         | Enforce at the tool/harness boundary                |
+| No loop / cost / rate cap                       | A hijacked or looping agent runs until it runs out of money   | Hard fail-closed kill-switches                      |
+| Redact PII only in the UI                       | The secret was already written to memory/logs                 | Redact before persistence, at the source            |

package/skills/shared/agent-safety/evals/README.md

@@ -0,0 +1,12 @@
+# Evals: agent-safety
+
+These cases are the trigger contract and capability rubric for the `agent-safety` skill.
+`should_trigger` and `should_not_trigger` assert that the description routes correctly —
+each negative names the real sibling it should route to instead (building-agents,
+secure-coding, agent-eval, prompt-engineering, cost-tracking). Run them through the repo's
+eval harness if one is wired up, or read them directly as the routing spec when reviewing
+the description. The `capability` block is not a pass/fail script: it is a rubric a human or
+an LLM-as-judge scores a sample guardrail design against — the design must cover every
+`must_include` item (deny-by-default tools, untrusted-content mediation, HITL on
+irreversible actions, memory hygiene, runtime caps) and map them to the OWASP Agentic Top
+10. This is a process/review skill, so there is no `verify.sh`; rigor lives in this eval.