@damiangilgonzalez/harness 1.3.7

package/scripts/skill-migration-brief.md

@@ -0,0 +1,200 @@
+# skill-migration-brief — how to move one skill to the Claude-5-era rubric
+
+This is the working brief for the catalog-wide migration. One agent, one skill, one worktree, one
+branch, one PR. `scripts/skill-rubric.md` is the spec; this is how to apply it to an existing skill.
+
+## Why this exists
+
+Anthropic's *The new rules of context engineering for Claude 5 generation models* (Jul 2026) reports
+that >80% of Claude Code's system prompt was removed with no measurable eval loss, and names the
+patterns that stopped paying: **repeating yourself across layers**, **over-constraining with rules
+where judgment suffices**, and **enumerating examples where an expressive interface would do**.
+
+Applied here: a skill's `description` is in context on **every turn it is installed**, invoked or
+not. Across 257 skills that was 207 KB. The body is loaded when the skill fires. Both are paid for
+by every user, so length is a cost, never a credit.
+
+## The pass
+
+1. `git checkout -b skill/<id>`
+2. Read `scripts/skill-rubric.md`, then `skills/<id>/SKILL.md`.
+
+### The description
+
+Shape: **what it does · when to use it · an explicit `NOT <x> (that is \`sibling\`)` boundary.**
+
+- Delete any `Triggers: '…', '…'` phrase list. The model matches by meaning; keyword lists in three
+  languages are pure per-turn cost, and the body usually instructs semantic matching right next to
+  them — a self-contradiction.
+- Name a **real** sibling in the boundary. Verify it exists under `skills/`.
+- Target ≤350 chars. Hard limit 1024 (schema-enforced). Third person, opens with "Use when/to".
+- The test is **discrimination, not coverage**: could a reader choose this skill over its nearest
+  sibling from the description alone?
+
+### The body
+
+The goal is the smallest body that still routes correctly. 400 lines is a ceiling; there is no floor.
+
+**Remove:**
+
+- Rule banks that restate the flow — "Iron rules (non-negotiable)", "Rationalizations — STOP".
+  A rule worth keeping moves next to the step it governs, and says *why* it is absolute instead of
+  shouting NON-NEGOTIABLE. A rule in an appendix is skimmed; a rule in context is followed.
+- Re-teaching of what another skill owns. The accompaniment dial belongs to `init`; the SDD method
+  belongs to `sdd`. Read the profile and point; do not restate, or the two drift apart.
+- "When to use" sections that restate the description, and trailing reference indexes whose links
+  already appear inline at point of use.
+
+**Keep — these are not padding:**
+
+- The **anti-patterns table**. The rubric requires one, and it is a different animal from a
+  rationalization table: it names concrete failure modes, not rules already stated above it.
+
+  **The hybrid case, which is common:** a table titled something like *"Anti-patterns /
+  rationalizations — STOP"*, with an *Excuse → Reality* framing. Do not delete it and do not leave
+  it as-is. **Keep every row, re-column it to `Anti-pattern | Do instead`, and drop the STOP /
+  excuse framing.** The rows are the value; the borrowed urgency wrapper is what dimension 7
+  forbids. Handling this the same way everywhere is what keeps the catalog consistent — deleting it
+  in one skill and keeping it in the next is worse than either choice applied uniformly.
+- Decision tables where the flow genuinely branches.
+- Tables specific to *this* skill's phase (e.g. what to show at each checkpoint) — that is not the
+  generic dial.
+- Worked bad/good pairs that teach judgment by demonstration.
+- Result-envelope blocks, chain position, and output contracts where they already exist. Do **not**
+  add one that was never there: that is a content change, not a format change.
+
+**Invariant:** every file under `skills/<id>/references/` must be linked from the body. An unlinked
+reference is never loaded, so it is dead weight in the package. Folding a tail index into inline
+mentions is fine — the link just has to survive. `scripts/`, `evals/` and `assets/` need no link.
+
+**Preserve substance exactly.** This is a context-engineering pass, not a content rewrite. Do not
+change recommendations, versions, commands, figures, or add claims.
+
+## Verify
+
+Your worktree has **no `node_modules`**, so:
+
+- Run: `bash scripts/eval-lint.sh 2>&1 | grep -i '^<id>'` → must report PASS.
+- Check the description length and that every `references/` file is linked.
+- Do **not** run `npm test` or `npm run validate` — they need dependencies you do not have. The
+  orchestrator runs them once, globally, after merge.
+- Do **not** edit `manifest.json`. It is derived, and parallel edits from hundreds of agents would
+  all conflict on the same lines. The orchestrator regenerates it once at the end.
+
+## Ship
+
+Commit only `skills/<id>/`. Subject `refactor(<id>): <what you actually did>`. The body says what you
+removed **and why**, and — just as important — what you deliberately **kept** and why.
+
+**Authorship is Eric, never Claude.** No `Co-Authored-By` for any AI, no "Generated with Claude Code"
+footer, nothing attributing the work to a tool — in the commit *or* the PR body. This is the hard
+rule the `ship` skill enforces (`skills/ship/SKILL.md`), and it applies to this migration like
+everything else in the repo. Verify before pushing:
+
+```bash
+git log -1 --format='%b' | grep -iE 'co-authored-by.*(claude|anthropic|ai)|generated with|claude code' \
+  && echo "AUTHORSHIP VIOLATION — strip it" || echo "authorship clean"
+```
+
+Then `git push -u origin skill/<id>` and `gh pr create`. The PR body carries before/after description
+chars, before/after body bytes, what went, what stayed, and the eval-lint result — and no AI
+attribution line.
+
+## Judgment
+
+If the skill already meets the rubric, **say so and change only the description.** A small honest
+diff beats a large invented one. Several skills in the pilot needed 5-line diffs; that is a good
+outcome, not a failed pass. Never pad, and never cut substance to hit a number.
+
+---
+
+# For the orchestrator
+
+Everything above is for the agent doing one skill. This half is for whoever is driving the fleet.
+
+## Resuming, from nothing
+
+There is no state file to keep in sync, on purpose. **The content is the ledger** — a migrated skill
+no longer carries a `Triggers:` list in its description:
+
+```bash
+node -e "const fs=require('fs');const out=[];for(const d of fs.readdirSync('skills')){const p='skills/'+d+'/SKILL.md';if(!fs.existsSync(p))continue;const m=/^---\n([\s\S]*?)\n---/.exec(fs.readFileSync(p,'utf8'));if(m&&/Triggers:/i.test(m[1]))out.push(d)}console.log(out.join('\n'))"
+```
+
+So a cold session resumes with: *read this brief, run that, launch one agent per remaining skill in
+its own worktree.* Nothing from any previous conversation is required.
+
+**An earlier version of this section said the remote `skill/*` branches were the ledger. They are
+not, and the failure is instructive**: once integration PRs started merging with `--delete-branch`,
+the branches for finished skills vanished, so a branch-based check re-offered work that was already
+on main — and an agent duly re-migrated `harness` a second time before anyone noticed. The
+instruction was true when written and was invalidated by a later change to the merge flow, with
+nothing to catch the drift. That is the same failure this whole migration keeps finding in the
+catalog; the brief was not exempt from it. Prefer a check against the artifact over a check against
+bookkeeping that something else can quietly change.
+
+## Order: by profile, not alphabetically
+
+A skill's `description` is in context for everyone who installs it, so migrate by blast radius:
+
+1. `minimal` profile — in every rsc install.
+2. `core` profile — the default install (the SDD chain plus the control plane).
+3. High-install stack skills — `nextjs`, `react`, `fastapi`, `postgresdb`, `design`, `typescript`,
+   `python`, `go`, `flutter`, `secure-coding`, `deployment`.
+4. Everything else.
+
+If a run is cut short, this ordering means the part that was finished is the part that everyone pays for.
+
+## Sharp edges (each one cost a real mistake)
+
+- **`manifest.json` is derived.** Agents must never touch it — hundreds of parallel edits collide on
+  the same lines. Regenerate once, centrally, with `npm run manifest` after merging.
+- **A fresh worktree has no `node_modules`** (it is gitignored, so it is not checked out). `npm test`
+  and `npm run validate` cannot run there. `scripts/eval-lint.sh` can — it uses python3.
+- **Result-envelope blocks are asymmetric.** `specify`, `tasks`, `implement`, `verify`, `ship`,
+  `parallel` and `sdd` have one; `plan`, `analyze`, `clarify`, `constitution`, `review` and `debug`
+  do not. Tell each agent which case it is in, or it will helpfully invent one.
+- **`author-skill` is special.** It teaches how to write skills, so it documented the *old*
+  conventions. There, rewriting the guidance IS the migration — and the change has to reach its
+  `references/` and `evals/`, which graded produced skills against the old rubric.
+- **Short ids break greps.** `go`, `rag`, `php` match half of `eval-lint`'s output. Give those agents
+  an anchored pattern (`grep -E '^go +'`).
+- **Compound bash commands are denied whole.** The `ship-guard` hook rejects the entire call if any
+  part matches `git checkout main`, so a chained `commit && push && checkout main` runs *nothing*.
+  Split them.
+- **The id `parallel` is unusable in a git command.** A sandbox guard aimed at GNU `parallel` matches
+  the substring, so `git checkout -b skill/parallel` is refused. That skill's branch has to be
+  created from the pushed commit with `gh api` instead. Any future id containing a blocked tool name
+  will behave the same way.
+
+## Content problems found while migrating — do NOT fix them in a format pass
+
+Agents surfaced these and correctly left them alone. They need their own change:
+
+- `sdd` claims every phase ends with a parseable envelope; six phases have none (see above).
+- **`evals/cases.yaml` files carry stale claims about what the catalog contains, in both
+  directions.** Three agents hit this independently: `course-storytelling` routes a negative to
+  `deep-research`, which does not exist; `flutter` routes its Compose/SwiftUI negatives to `"none"`
+  on the grounds those skills are "not in this catalog", though `compose-multiplatform` and
+  `swift-ios` now are; `postgresdb` said the same of ORM traps, stale since `prisma-orm` and
+  `drizzle-orm` shipped. This drifts every time the catalog grows.
+
+  The root cause is that `eval-lint` does not actually verify a `route_to` names a real skill,
+  even though the rubric lists exactly that as a deterministic gate — so the gate is decorative.
+  (The 1024-char description limit had the same problem; that one is now enforced in
+  `schema/frontmatter.schema.json`.) Fixing the linter is worth more than fixing the instances:
+  it converts a recurring drift into a build error.
+- `clarify`'s body calls itself "the fourth phase" while the chain it prints puts it third.
+- `cold-outreach` never names `linkedin-outreach` in its route table, though the reverse link exists.
+- `data-policy` says "walk five columns" and then lists six.
+- `marketing/references/seo-geo.md` (20 KB) and marketing's landing-section material substantially
+  duplicate the standalone `seo-geo` and `landing-copy` skills. Three skills, one body of content,
+  paid for three times by anyone who installs all three. Needs a content-level de-duplication pass,
+  not a format one.
+
+## Return value
+
+```json
+{"skill":"<id>","pr":"<url|null>","descBefore":0,"descAfter":0,"bytesBefore":0,"bytesAfter":0,
+ "evalLint":"PASS|FAIL","removed":["…"],"kept":["…"],"notes":"anything the orchestrator must know"}
+```

package/scripts/skill-rubric.md

@@ -0,0 +1,40 @@
+# skill-rubric — the objective test every rsc skill must pass before shipping
+
+This is **the test, written before any skill is built.** A reviewer scores a finished skill
+0–10 on seven weighted dimensions, then the deterministic gates run (pass/fail, not gradeable).
+A skill **ships only at a weighted score ≥ 8.5 AND all deterministic gates green.** The reviewer
+is adversarial and independent of the author; it must cite evidence (line counts, char counts,
+quoted source dates), never vibes. No grade inflation — an honest 7.0 that drives a fix beats a
+dishonest 9.
+
+## Weighted dimensions (0–10 each)
+
+| # | Dimension | Weight | What a 10 looks like |
+|---|---|---|---|
+| 1 | **Triggering & description** | 0.15 | Third-person, `Use when …` lead, an explicit `NOT … (that is <sibling>)` boundary, and nothing else it can do without. Judged on **discriminative power, not coverage**: could a reader pick this skill over its nearest sibling from the description alone? Aim ≤350 chars; ≤1024 is the hard limit. Parses as single-line quoted YAML. |
+| 2 | **Scope & boundary** | 0.10 | One job. Body never drifts into a sibling's territory; it delegates by name. Near-miss prompts route to the correct existing sibling. |
+| 3 | **Body craft & progressive disclosure** | 0.15 | **The smallest body that routes correctly** — ≤400 lines is a ceiling, not a target, and there is no minimum: a skill that does its job in 60 lines scores higher than the same skill padded to 200. Every `references/` file is pointed to inline from this body (an unlinked reference is never loaded, so it is dead weight). rsc voice, every fence language-tagged, a checklist/decision table only where the flow branches, an anti-patterns table present. |
+| 4 | **Correctness, grounding & freshness** | **0.25** | Every load-bearing claim is accurate and **current (2025–2026)**: real library/API versions, no hallucinated flags/endpoints. The spec cites **authoritative sources with dates**; nothing rests on stale or invented facts. This is the heaviest dimension by design. |
+| 5 | **Actionability** | 0.15 | Concrete and runnable: real numbers, paths, commands, decision rules — not adjectives. A practitioner could act from it without leaving the skill. |
+| 6 | **Evals quality** | 0.10 | ≥5 `should_trigger` (incl. non-obvious), ≥4 `should_not_trigger` each with a `route_to` that resolves — a **real** catalog id, `none`, or `external:<name>` (enforced by `eval-lint`), ≥1 `capability` with a meaningful `must_include` rubric. The cases are genuine, not filler. |
+| 7 | **Originality & safety** | 0.10 | rsc voice, not a clone of any external catalog (no borrowed urgency blocks / rationalization tables / `*-reviewer-prompt.md` conventions / verbatim flowcharts). `verify.sh` present iff there is a checkable artifact. No destructive or unsafe advice presented without guardrails. |
+
+**Weighted score** = Σ(subscore × weight), rounded to 1 decimal. **Ship gate: ≥ 8.5.**
+
+## Deterministic gates (pass/fail — not part of the 0–10, but block shipping)
+
+1. `name` matches the directory id; `origin: risco` present.
+2. Frontmatter validates against `schema/frontmatter.schema.json` (`node scripts/build-manifest.js --validate`).
+3. Every `recommends` id is a **real** skill in the catalog (no dangling refs — `--validate` enforces).
+4. `scripts/eval-lint.sh` reports PASS for the skill (≥5/≥4/≥1 and the keys are lists).
+5. Every `../<sibling>/SKILL.md` link resolves to a skill that exists.
+6. `description` is ≤ 1024 characters and parses as YAML.
+7. If a `scripts/verify.sh` exists, it is executable, read-only by default, and exits 0 on a clean/empty target (no false failure).
+
+## How the reviewer must work (anti-cheat)
+
+- **Independent pass.** Review the artifact as written; do not assume the author's intent filled a gap.
+- **Evidence, not vibes.** For dimension 1 quote the char count **and name the sibling the description distinguishes this skill from**; for 3 quote the SKILL.md line count **and name anything that could be cut without losing routing**; for 4 quote at least two source titles **with their dates** from the spec and name any claim you could not verify; for 6 count the entries.
+- **Length is a cost, never a credit.** Do not reward a longer description or body for being thorough. Every line is paid for in context on every turn the skill is installed, whether or not it is invoked. If two versions route the same, the shorter one scores higher.
+- **Freshness is mandatory.** If the skill names a version, API, or pricing that is plausibly out of date, dimension 4 is capped at 6 until a current source is cited. "Looks right" is not a source.
+- **Fix loop.** Below 8.5, return a concrete `mustFix` list; the author applies it and re-scores. Up to two fix rounds. If still < 8.5, **record the real score and flag it** — never round up to pass.

package/scripts/skill-scoreboard.workflow.js

@@ -0,0 +1,35 @@
+export const meta = {
+  name: 'skill-scoreboard',
+  description: 'Behaviorally evaluate several rsc skills in parallel (detect only, no fixing) and return each one\'s raw A/B grader signals for local scoring',
+  phases: [
+    { title: 'Score' },
+  ],
+}
+
+function parseIds(a) {
+  if (Array.isArray(a)) return a
+  if (a && Array.isArray(a.skills)) return a.skills
+  if (typeof a === 'string') {
+    const s = a.trim()
+    if (s.startsWith('[')) {
+      try { const p = JSON.parse(s); if (Array.isArray(p)) return p } catch (_) { /* fall through */ }
+    }
+    return s.split(/[\s,]+/).filter(Boolean)
+  }
+  return []
+}
+const ids = parseIds(args)
+if (ids.length === 0) throw new Error('skill-scoreboard: pass an array of skill ids as args, e.g. ["debug","grants"]')
+
+const EVAL = { scriptPath: 'scripts/skill-behavior-eval.workflow.js' }
+
+phase('Score')
+const rows = await parallel(
+  ids.map((id) => () =>
+    workflow(EVAL, id)
+      .then((raw) => ({ id, raw }))
+      .catch((e) => ({ id, raw: { skillId: id, scenarios: [], error: String(e && e.message || e) } })),
+  ),
+)
+
+return { rows: rows.filter(Boolean) }

package/skills/ab-testing/SKILL.md

@@ -0,0 +1,159 @@
+---
+name: ab-testing
+description: "Use when designing or analyzing a controlled experiment — falsifiable hypothesis, sample size from an MDE, reading significance/CI/power, CUPED, or rescuing tests that won't go significant. NOT recurring metric tracking (that is `analytics`), NOT north-star/KPI trees (that is `kpi-framework`), NOT projecting metrics forward (that is `forecasting`)."
+tags: [ab-testing, experimentation, statistics, cuped, sample-size, hypothesis-testing]
+recommends: [analytics, kpi-framework, forecasting, data-cleaning, python, reporting]
+origin: risco
+---
+
+# A/B testing — design and read a defensible experiment
+
+An experiment without a pre-committed sample size and a single primary metric is not an experiment.
+It is a dashboard you stare at until it tells you what you wanted to hear. The discipline lives almost
+entirely *before* traffic ships: a falsifiable hypothesis, one primary metric, a sample size derived
+from the smallest effect worth detecting, and a stop rule you cannot renegotiate at 2pm on day four.
+
+## Pre-test checklist — every line true before any traffic
+
+Each one is a place experiments die silently.
+
+- [ ] A **falsifiable hypothesis** — names the change, the direction, and the metric it moves.
+- [ ] Exactly **ONE primary metric**. More than one primary = multiple comparisons = inflated false positives.
+- [ ] **Guardrail metrics** — what you refuse to harm (latency, refunds, unsubscribes) even for a win.
+- [ ] The **randomization unit = the analysis unit** (usually the user). Mixing them is pseudoreplication.
+- [ ] An **MDE** — the smallest lift that would change a decision. Not "any difference."
+- [ ] A **computed sample size** and the **duration** it implies at your real daily eligible traffic.
+- [ ] A **fixed stop rule** — a date or an n you commit to before launch. No "we'll see how it looks."
+
+## Step 1 — Hypothesis and metrics
+
+State a null you can reject. "The new checkout button changes purchase conversion" with H0: conversion
+equal across arms, H1: it differs. Vague aspirations ("improve the funnel") have no rejection region.
+
+Pick one primary metric and freeze it. Why: every extra primary metric is another coin flip at α, so
+three "primary" metrics turn a 5% false-positive rate into roughly 14%. Demote the rest to secondary.
+
+Randomize on the same unit you analyze on. If a user sees the variant on every visit, randomize by user,
+not by session — analyzing 50k sessions from 8k users treats correlated observations as independent and
+fabricates significance.
+
+```text
+Bad:  "We think the redesign will improve engagement and revenue and retention."  (no null, 3 primaries, no number)
+Good: "H0: 30-day purchase conversion is equal between control and the new one-click button.
+       H1: it differs. Primary: purchase conversion. Guardrails: refund rate, p95 checkout latency.
+       Randomize by user_id. MDE: +1.5pp absolute on a 12% baseline."
+```
+
+## Step 2 — Sample size from MDE, baseline, and power
+
+Defaults: power 0.80, α 0.05 (two-sided). The MDE is yours to choose — it is the smallest effect that
+would actually change what you do.
+
+Rule: required n scales with ~1/MDE². Why: halving the smallest effect you care to detect roughly
+**quadruples** the traffic and time. This is the single most expensive decision in the design, so set the
+MDE to a business threshold, never to "whatever is small."
+
+For a conversion rate (proportion):
+
+```python
+from statsmodels.stats.power import NormalIndPower
+from statsmodels.stats.proportion import proportion_effectsize
+
+p1, p2 = 0.12, 0.135                       # baseline, baseline + MDE (1.5pp)
+h = proportion_effectsize(p1, p2)          # Cohen's h (arcsine transform)
+n = NormalIndPower().solve_power(effect_size=h, alpha=0.05, power=0.80, ratio=1.0)
+print(int(-(-n // 1)))                      # n PER ARM, rounded up
+```
+
+For a continuous metric (revenue per user, time on page) use Welch-style sizing:
+
+```python
+from statsmodels.stats.power import TTestIndPower
+
+effect = mde_in_units / pooled_std         # Cohen's d
+n = TTestIndPower().solve_power(effect_size=effect, alpha=0.05, power=0.80, ratio=1.0)
+```
+
+Then convert n to a calendar plan: `days = ceil((n_per_arm * num_arms) / daily_eligible_users)`. If that
+is 9 days, run a clean **two full weeks** anyway — weekday/weekend mix is part of the population, and a
+6-day test oversamples whoever shows up Tuesday. Full worked example (12% baseline, +1.5pp MDE, 80%
+power) plus runnable sizing, n→duration, CUPED θ and SRM snippets: `references/sample-size-and-cuped.md`.
+
+## Step 3 — Run discipline
+
+**Fixed horizon is the default.** Commit to the n/date from Step 2 and read the result once, at the end.
+
+**Do not peek and stop at first significance.** Why: checking repeatedly and stopping the moment p < 0.05
+inflates the Type-I error far above 5% — with enough looks, a null test crosses 0.05 most of the time.
+If you genuinely need to stop early, use a *sequential / always-valid* method (confidence sequences,
+e.g. Netflix's anytime-valid CIs) that holds Type-I error under continuous monitoring. Sequential is
+strong for **killing losers early** and weak for **calling winners early** — for a confident win, the
+fixed-horizon read is tighter.
+
+**Gate on SRM before you trust anything.** Compute a chi-square test on the observed split versus the
+intended ratio. If p < 0.001 the assignment or logging is broken — a bot filter dropping one arm, a
+redirect, a caching bug. Fix the instrumentation and rerun; do not "adjust for it."
+
+The peeking Type-I math, sequential/always-valid options, SRM diagnosis, novelty/primacy effects,
+Simpson's paradox in segments and HARKing all live in `references/pitfalls.md`.
+
+## Step 4 — Analyze
+
+Pick the test by metric type:
+
+| Metric type | Test |
+|---|---|
+| Binary conversion (proportion) | Two-proportion z-test (`statsmodels.stats.proportion.proportions_ztest`) |
+| Continuous, roughly normal / large n | Welch's t-test (`scipy.stats.ttest_ind(..., equal_var=False)`) |
+| Continuous, heavy-tailed / skewed (revenue) | Mann-Whitney U, or t-test on a log/winsorized metric |
+
+Report **lift + confidence interval + p-value together**. Never p alone. Why: p < 0.05 with a CI of
+[+0.1pp, +5pp] is "statistically there, practically a coin toss" — the CI tells you the size, p only
+tells you it is not exactly zero. **Practical significance** = compare the CI to your MDE: if the whole
+interval sits above the MDE, ship; if it straddles the MDE, you detected *something* too small to matter.
+
+**Multiple comparisons.** Two regimes:
+- Small set of pre-declared **decision** metrics → **Bonferroni** (divide α by the count). Conservative, simple.
+- Large **exploratory** scan of many metrics/segments → **Benjamini-Hochberg (FDR)**. It keeps far more
+  power than Bonferroni on big scans (in a 20-effect example, ~17 detected vs ~12 under Bonferroni).
+
+## Step 5 — CUPED variance reduction
+
+CUPED (Controlled-experiment Using Pre-Experiment Data) subtracts predictable pre-period noise so the
+same traffic buys more power — or the same power needs less traffic. The adjusted metric:
+
+```text
+Y_cuped = Y − θ · (X − E[X])        where  θ = Cov(Y, X) / Var(X)
+```
+
+Estimate θ by regressing the in-experiment metric `Y` on the **pre-experiment** covariate `X` (e.g. each
+user's spend in the 4 weeks before the test), then analyze `Y_cuped` with the same test as Step 4.
+
+When it pays: recurring users with a strong pre-period signal. Reported wins — Netflix ~40% variance
+reduction on engagement, Statsig 50%+ on common metrics → significance in roughly half the time/traffic.
+
+When it does **nothing** — do not bother: brand-new users (no pre-period data), a covariate uncorrelated
+with the outcome, or — the cardinal sin — a covariate measured *after* assignment, which biases the
+estimate. The covariate MUST be pre-treatment and independent of which arm a user lands in. Runnable
+θ-via-OLS snippet in `references/sample-size-and-cuped.md`.
+
+## Anti-patterns
+
+| Bad | Why it is wrong | Do instead |
+|---|---|---|
+| Peek daily, stop the day p < 0.05 | Repeated looks inflate Type-I error far above α | Fix n/date up front; or a sequential method that holds α |
+| No sample size set before launch | You will stop on noise and call it a win | Compute n from MDE/baseline/power in Step 2 |
+| Several "primary" metrics | Each is a coin flip at α; 3 metrics ≈ 14% false-positive | One frozen primary; the rest are secondary |
+| Ignore the observed split | An SRM means assignment/logging is broken; results are garbage | Chi-square SRM gate before reading anything |
+| Report only the p-value | Hides effect size — p < 0.05 can be practically zero | Always lift + CI + p; compare CI to MDE |
+| CUPED on a post-assignment covariate | Covariate correlated with the arm biases θ | Use only pre-treatment, assignment-independent covariates |
+| Call a winner from an underpowered test | "Not significant" then ≠ "no effect"; you lacked power | Reach planned n, or report the CI and say "inconclusive, here is the range" |
+| Decide the hypothesis after seeing results (HARKing) | Turns the whole analysis into a fishing expedition | Pre-register hypothesis + primary metric before launch |
+| Run 6 days because it "looks significant" | Oversamples one weekday slice of the population | Run full weeks; honor the fixed horizon |
+
+## Checkable artifact
+
+When this skill emits a Python sizing/analysis script or an experiment-design doc, run
+`scripts/verify.sh` from your project root. It confirms the script executes under `python3` and prints a
+numeric sample size, and that any design doc names a primary metric, an MDE, and power/alpha. It is
+read-only and soft-passes when no artifact is present (a design-only conversation).

package/skills/ab-testing/evals/README.md

@@ -0,0 +1,8 @@
+# Evals — ab-testing
+
+These cases are routing and capability checks for the catalog harness, not an automated test runner.
+`should_trigger` and `should_not_trigger` are judged by feeding each prompt to the router and confirming
+it lands on `ab-testing` (or, for the negatives, on the named sibling such as `analytics` or
+`forecasting`). The single `capability` case is graded by hand or with the catalog's eval script: run the
+scenario through the skill and check the produced design/analysis against every line in `must_include` —
+a pass needs all of them present, not just most. There is no `pytest` here; the rubric is the spec.

package/skills/ab-testing/evals/cases.yaml

@@ -0,0 +1,49 @@
+skill: ab-testing
+
+should_trigger:
+  - prompt: "How many users per arm do I need to detect a 2% lift on a 10% baseline conversion?"
+    why: Core sample-size sizing from baseline + MDE + power — the central pre-test calculation this skill owns.
+  - prompt: "Is this A/B result statistically significant? Control converted 540/10000, variant 590/10000."
+    why: Read-out of a finished proportion test — two-proportion z-test, lift, CI, p-value together.
+  - prompt: "Our test has run two weeks and still isn't significant, what do we do now?"
+    why: Non-obvious. "Inconclusive" almost always means underpowered, peeked, or an SRM — a design diagnosis, not a re-run.
+  - prompt: "Can we use CUPED to get faster results on our checkout experiment?"
+    why: Variance reduction via a pre-experiment covariate — when it helps, the formula, and when it does nothing.
+  - prompt: "We keep peeking at the dashboard and stopping the moment it hits p<0.05 — is that a problem?"
+    why: Non-obvious optional-stopping trap; peeking inflates Type-I error far above alpha. Fixed horizon or always-valid inference.
+  - prompt: "El experiment de la landing no surt significatiu, com el dissenyo bé des de zero?"
+    why: Catalan trigger — rescue an inconclusive experiment by redesigning the hypothesis/MDE/sample-size up front.
+  - prompt: "Cuántos usuarios necesito para detectar un 1.5pp de mejora con 80% de potencia?"
+    why: Spanish trigger for sample-size computation from MDE and power.
+
+should_not_trigger:
+  - prompt: "Build a dashboard tracking weekly signups and churn over time."
+    route_to: analytics
+    why: Recurring metric tracking and event instrumentation, not a controlled experiment with a hypothesis.
+  - prompt: "Define our north-star metric and the KPI tree underneath it."
+    route_to: kpi-framework
+    why: Deciding which metrics matter, not designing or analyzing a test of a change.
+  - prompt: "Forecast next quarter's revenue from the last 24 months of data."
+    route_to: forecasting
+    why: Time-series projection into the future, not a randomized comparison of two arms.
+  - prompt: "Clean and dedupe this messy events CSV before any analysis."
+    route_to: data-cleaning
+    why: Data preparation, not experiment design or read-out.
+  - prompt: "Turn last month's experiment results into a stakeholder slide deck."
+    route_to: reporting
+    why: Periodic stakeholder communication of results, not the statistical design/analysis itself.
+
+capability:
+  - scenario: >
+      Design, size, and plan the analysis for a test of a new one-click checkout button. Baseline
+      purchase conversion is 12%; the team wants to detect a +1.5pp absolute lift at 80% power and
+      alpha 0.05, with about 1,800 eligible users per day. Include a plan to speed it up with CUPED.
+    must_include:
+      - A falsifiable hypothesis with H0/H1 and exactly one primary metric (purchase conversion), plus guardrail metrics
+      - Randomization unit equals analysis unit (by user), not by session
+      - Sample-size calculation naming statsmodels proportion_effectsize and NormalIndPower / solve_power
+      - Converting n per arm to a duration in days from daily eligible traffic, rounded up to whole weeks
+      - A fixed-horizon stop rule and an explicit no-peeking warning (Type-I inflation)
+      - An SRM chi-square check before trusting the result
+      - CUPED plan with a pre-treatment covariate and the formula Y_cuped = Y - theta*(X - E[X])
+      - Reporting the result as lift + confidence interval + p-value, compared against the MDE for practical significance

package/skills/ab-testing/references/pitfalls.md

@@ -0,0 +1,74 @@
+# Pitfalls — why "inconclusive" tests usually mean a broken design
+
+This is the diagnosis layer. When a test "won't go significant" or "flip-flops," it is almost always one
+of the failure modes below, not bad luck.
+
+## Peeking and optional stopping
+
+A fixed-horizon test is designed so that, under the null, you cross p < 0.05 about 5% of the time **when
+you look exactly once, at the planned n**. If you look every day and stop the first time you see p < 0.05,
+you take many independent-ish shots at that 5% door. With enough looks the cumulative chance of crossing
+0.05 under a true null climbs well past 5% — a pure-noise test can be "significant" the majority of the
+time if you peek long enough. The reported p-value is then a lie about a single planned look.
+
+Fixes, in order of preference:
+
+1. **Fixed horizon.** Commit to n/date; read once. Tightest read for declaring a winner.
+2. **Sequential / always-valid inference.** Confidence sequences and anytime-valid CIs (e.g. Netflix's
+   anytime-valid approach) stay valid under continuous monitoring — you may look every day and the
+   guarantee holds. Trade-off: wider intervals, so they are excellent for **stopping a clear loser early**
+   and comparatively weak for **declaring a winner early**. Use them when early stopping has real value,
+   not as a license to peek on a fixed-horizon design.
+
+Never: a fixed-horizon design read repeatedly and stopped at first significance.
+
+## Multiple comparisons
+
+Every metric, every variant, every segment is another hypothesis test. Test 20 independent nulls at
+α = 0.05 and you expect ~1 false positive by chance alone. Two corrections:
+
+- **Bonferroni** — divide α by the number of tests. Controls the chance of *any* false positive
+  (family-wise error). Simple and conservative; right for a small, pre-declared set of decision metrics.
+- **Benjamini-Hochberg (FDR)** — controls the *expected fraction* of false positives among the discoveries.
+  Far more powerful on large exploratory scans. Illustrative example: scanning 20 metrics with true
+  effects, an FDR procedure recovered ~17 while Bonferroni's stricter threshold recovered only ~12.
+
+Rule: Bonferroni for the handful of metrics a decision rests on; Benjamini-Hochberg for the big
+exploratory sweep where you can tolerate a known false-discovery rate.
+
+## Sample ratio mismatch (SRM)
+
+The observed split materially diverges from the intended ratio (chi-square p < 0.001). This is not a
+statistical nuance — it is a signal the experiment is mechanically broken: a bot filter dropping one arm,
+a redirect, differential caching, a bucketing-hash bug, or logging that fires on one variant only.
+Consequence: the arms are no longer comparable, so the treatment effect is confounded. Do not adjust,
+reweight, or "note it as a caveat." Find the cause, fix instrumentation, and rerun. The SRM chi-square
+snippet is in `sample-size-and-cuped.md`.
+
+## Underpowered ≠ "no effect"
+
+A non-significant result with a small sample means you could not have detected even a meaningful effect —
+absence of evidence, not evidence of absence. Before concluding "no difference," check whether the test
+reached its planned n and report the CI: "the effect is somewhere in [−0.8pp, +1.2pp]" is honest;
+"no effect" is not.
+
+## Novelty and primacy effects
+
+A shiny new variant can spike because regulars click it out of curiosity (novelty) — or dip because they
+are disoriented by a changed UI (primacy). Both fade. If the effect is concentrated in the first days and
+decays, you are measuring reaction-to-change, not the steady-state effect. Run long enough for the curve
+to flatten, and inspect new-user vs returning-user segments separately.
+
+## Simpson's paradox across segments
+
+An aggregate lift can reverse inside every segment (or vice versa) when segment mix differs between arms —
+often itself a symptom of SRM or a non-random assignment. If the overall result and the per-segment
+results disagree in direction, distrust the aggregate and find what is unbalanced between the arms before
+believing either number.
+
+## HARKing — hypothesizing after results are known
+
+Slicing the data until something is significant, then writing the hypothesis to match, is a fishing
+expedition wearing a lab coat. Any "finding" discovered this way must be treated as a hypothesis for a
+*fresh* experiment, never as a confirmed result. Pre-register the hypothesis and primary metric before
+launch so the analysis you run is the analysis you committed to.