@damiangilgonzalez/harness 1.3.7

package/skills/data-scraper/references/frameworks.md

@@ -0,0 +1,116 @@
+# Frameworks reference
+
+Copy-paste resilient starters. Match the tool to the site profile from the picker in `../SKILL.md`. Every starter assumes the legal gate already passed.
+
+## Crawlee (Python) — resilient recurring crawler
+
+Default for recurring jobs that must not break. The `RequestQueue` is disk-persisted, so a crash resumes from where it stopped. Per-host concurrency and pacing are config, not hand-rolled.
+
+```python
+import asyncio
+from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
+
+crawler = PlaywrightCrawler(
+    max_requests_per_crawl=10_000,
+    max_request_retries=4,                 # retries use backoff internally
+    concurrency_settings={"max_concurrency": 4},  # cap per run; pace per host below
+    request_handler_timeout_secs=45,
+)
+
+@crawler.router.default_handler
+async def handle(ctx: PlaywrightCrawlingContext) -> None:
+    title = await ctx.page.text_content('[data-testid="listing-card"] h2')
+    if not title:
+        raise LookupError(f"required field 'title' missing at {ctx.request.url}")  # fail loud
+    await ctx.push_data({                  # idempotent: keyed on url downstream
+        "url": ctx.request.url,
+        "title": title.strip(),
+    })
+    await ctx.enqueue_links(selector='a[href^="/listing/"]')  # follow only listing links
+
+async def main() -> None:
+    # seed from sitemap, not a blind crawl
+    await crawler.run(["https://example.com/sitemap.xml"])
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+Why Crawlee for new resilient work: it wraps Playwright, rotates proxies, ships browserforge fingerprints, and the queue resumes after a crash — all the resilience patterns from `../SKILL.md` come built in instead of hand-rolled.
+
+## Playwright direct — one-off SPA scrape
+
+When the job is one-shot and a full crawler is overkill, but the data only exists after JS runs.
+
+```python
+import asyncio, random
+from playwright.async_api import async_playwright
+
+async def scrape(urls: list[str]) -> list[dict]:
+    out = []
+    async with async_playwright() as p:
+        browser = await p.chromium.launch(headless=True)
+        ctx = await browser.new_context(locale="en-US")
+        for url in urls:
+            page = await ctx.new_page()
+            await page.goto(url, wait_until="networkidle", timeout=30_000)
+            title = await page.text_content('[data-testid="listing-card"] h2')
+            if not title:
+                raise LookupError(f"required field 'title' missing at {url}")
+            out.append({"url": url, "title": title.strip()})
+            await page.close()
+            await asyncio.sleep(random.uniform(1.0, 3.0))  # 1-3s per host, jittered
+        await browser.close()
+    return out
+```
+
+## Scrapy — pure-Python static, AUTOTHROTTLE
+
+Legacy/static targets. AUTOTHROTTLE adapts the delay to the host's response latency instead of a fixed sleep.
+
+```python
+# settings.py
+ROBOTSTXT_OBEY = True               # honor robots.txt, non-negotiable
+AUTOTHROTTLE_ENABLED = True
+AUTOTHROTTLE_START_DELAY = 1.0      # seconds
+AUTOTHROTTLE_MAX_DELAY = 10.0
+AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0   # keep it low, per host
+CONCURRENT_REQUESTS_PER_DOMAIN = 4
+DOWNLOAD_DELAY = 1.0
+RETRY_ENABLED = True
+RETRY_TIMES = 3                     # Scrapy backs off between retries
+DEFAULT_REQUEST_HEADERS = {"Accept-Language": "en-US,en;q=0.9"}
+HTTPCACHE_ENABLED = True            # conditional-GET style caching on re-crawl
+```
+
+Pick Scrapy only for an existing Scrapy codebase or pure static targets — its Twisted core lags the asyncio ecosystem, so new resilient work goes to Crawlee.
+
+## httpx + selectolax — static, no fingerprint wall
+
+Cheapest path when the HTML is server-rendered and the host does not fingerprint.
+
+```python
+import time, random, httpx
+from selectolax.parser import HTMLParser
+
+def scrape(urls: list[str], headers: dict) -> list[dict]:
+    out = []
+    with httpx.Client(headers=headers, timeout=20, follow_redirects=True) as c:
+        for url in urls:
+            r = c.get(url)
+            if r.status_code == 429:                       # honor Retry-After
+                time.sleep(int(r.headers.get("Retry-After", "30")))
+                r = c.get(url)
+            r.raise_for_status()
+            tree = HTMLParser(r.text)
+            node = tree.css_first('[data-testid="listing-card"] h2')
+            if node is None:
+                raise LookupError(f"required field 'title' missing at {url}")
+            out.append({"url": url, "title": node.text(strip=True)})
+            time.sleep(random.uniform(1.0, 3.0))           # 1-3s per host
+    return out
+```
+
+If this starts returning 403 while a browser returns 200, swap `httpx` for `curl_cffi` with `impersonate="chrome131"` (see `anti-bot.md`).
+
+Sources: crawlee.dev/python changelog; npmjs.com/package/playwright & pypi.org/project/playwright (1.60.0; 1.59 shipped 2026-04-01); use-apify.com "Crawlee vs Scrapy vs BeautifulSoup 2026". All accessed 2026-06-02.

package/skills/data-scraper/references/legal-compliance.md

@@ -0,0 +1,64 @@
+# Legal & compliance reference
+
+Depth behind the legal gate in `../SKILL.md`. You are not a lawyer and you never claim to be — you document the basis and flag where a human must sign off.
+
+## robots.txt and ai.txt
+
+Fetch both before queuing URLs. `robots.txt` states crawler preferences by user-agent and path; `ai.txt` (and the `noai` / `noimageai` signals some sites publish) is the emerging objection channel for AI training use specifically.
+
+**Neither is law.** robots.txt is a voluntary convention from 1994 with no statutory force — no jurisdiction makes a `Disallow` binding by itself, and courts treat it as evidence of the parties' expectations, not as the thing you violated. So it is input to your decision, never a hard stop. What *is* binding sits in the sections below: an accepted ToS, data-protection law, and anti-circumvention.
+
+```python
+import urllib.robotparser, urllib.parse
+
+def allowed(url: str, ua: str = "my-crawler") -> bool:
+    base = "{0.scheme}://{0.netloc}".format(urllib.parse.urlparse(url))
+    rp = urllib.robotparser.RobotFileParser()
+    rp.set_url(base + "/robots.txt")
+    rp.read()                      # network call; cache per host
+    return rp.can_fetch(ua, url)   # False -> do not queue this URL
+```
+
+Rules:
+
+Working defaults — deviate consciously, not by accident:
+
+- A `Disallow` for your path is the host's stated preference. Honoring it is the default and costs you little; overriding it is a judgment call you record, not a rule you break. Scraping your own property, a site you have written permission for, or public pages for research and archiving are all ordinary reasons to override.
+- Respect `Crawl-delay` if present — it is the host telling you its tolerance, and the cheapest way to not get banned.
+- An `ai.txt` / `noai` objection means the site objects to AI-training use. Honoring it is the cleanest evidence of good faith and matters most where the use is exactly what was objected to.
+- Whatever you decide, log it with the scrape. "We read robots, disallowed `/search`, proceeded anyway because X" is a defensible record; silence is not.
+
+## ToS red flags
+
+Read the terms before scraping. The presence of any of these moves you toward **narrow** or **stop**:
+
+- An explicit "no automated access / no scraping / no crawling" clause.
+- Acceptance gated by **login** — accepting at sign-in is the contract hook that made *Meta v. Bright Data* a breach-of-contract case rather than a CFAA one.
+- A clause assigning IP/database rights over the listings themselves (common on directories and aggregators).
+- An API offered under separate commercial terms — using it routes you to `../api-connector-builder/SKILL.md` and scraping around it looks like circumvention.
+
+## GDPR lawful-basis decision flow
+
+Personal data = names, emails, photos, reviews, IP addresses, anything identifying a natural person. If the scrape touches any:
+
+1. **New purpose check.** Is your purpose the same as why the data was published? Reusing public personal data for aggregation, resale, or AI training is a *new* purpose and a severe breach risk — fines reach tens of millions of EUR.
+2. **Lawful basis.** Usually **legitimate interest** for public-data scraping — but it requires a balancing test (your interest vs. the person's reasonable expectations). Document it. See `../gdpr-privacy/SKILL.md` for the LIA.
+3. **Minimization.** Collect only the fields the purpose needs. Drop everything else at extraction, not later.
+4. **Special categories.** Health, politics, religion, sexual orientation, biometrics (Art. 9) — filter these out at the source. Do not collect them speculatively.
+5. **Objection signals.** Honor robots.txt / ai.txt objections as evidence of respecting data-subject expectations.
+
+## Minimization & retention checklist
+
+- [ ] Field list maps 1:1 to the stated purpose — no "just in case" columns.
+- [ ] Special categories filtered at extraction.
+- [ ] Retention period set, with a deletion trigger (not "keep forever").
+- [ ] Source URL + scrape timestamp recorded per record for provenance.
+- [ ] A human owner signed off on the basis if any personal data is involved.
+
+## Case summaries (2024-2025)
+
+- **hiQ v. LinkedIn** — scraping *public* data is not automatically a CFAA violation. The floor, not a license; everything else (auth, contract, GDPR) still applies.
+- **Meta v. Bright Data (2024)** — the exposure is **breach of contract** when you accepted ToS, typically by logging in. Logged-out public scraping weakens the contract claim. Lesson: stay logged-out, never bypass auth.
+- **Reddit v. Perplexity AI (2025)** — centers on whether **rate limits and anti-bot measures were circumvented**, distinct from the public-vs-private axis. Bypassing a control you were shown (CAPTCHA, hard block, enforced limit) is materially worse than pacing public pages. Lesson: pace, do not circumvent.
+
+Sources: groupbwt.com "Web Scraping Legal Issues: 2025 Enterprise Compliance Guide"; sociavault.com court-case roundup; promptcloud.com 2026 compliance guide; medium.com/deep-tech-insights "The €20 Million GDPR Mistake" (2025); zyte.com AI personal-data scraping guidance. All accessed 2026-06-02.

package/skills/data-scraper/scripts/verify.sh

@@ -0,0 +1,166 @@
+#!/usr/bin/env bash
+#
+# verify.sh — compliance/resilience linter for the `data-scraper` skill.
+#
+# WHAT IT DOES (read-only; never edits a file)
+#   Statically lints a candidate scraper (a file or directory of source) for the
+#   four properties the skill insists on before a scrape ships. It greps source
+#   for signal patterns — it does not run the scraper or hit any network.
+#
+#   Per-file scan of *.py *.js *.ts *.mjs *.cjs (the languages the starters use).
+#   A file with no scraping signal at all is skipped, so this never false-fails
+#   on an unrelated repo (and an empty/clean target exits 0).
+#
+#   Checks (a FAILURE is only ever an unbounded scrape or a circumvention call —
+#   those are not judgement calls):
+#     1. Compliance preflight present  -> robots.txt / ToS / ai.txt / robotparser
+#        / ROBOTSTXT_OBEY reference somewhere in the target.   (missing -> warn)
+#     2. Per-host pacing present       -> a delay/sleep/throttle/concurrency/
+#        rate-limit/Crawl-delay/AUTOTHROTTLE signal.           (missing -> FAIL)
+#     3. Retry / backoff present       -> backoff/retry/Retry-After/max_retries
+#        /exponential signal.                                  (missing -> warn)
+#     4. No circumvention              -> a CAPTCHA-solver call (2captcha,
+#        twocaptcha, anticaptcha, capsolver, deathbycaptcha, solve_captcha) or an obvious
+#        auth-bypass marker.                                   (present -> FAIL)
+#
+#   "Scraping signal" = the file imports/uses one of: requests, httpx, curl_cffi,
+#   playwright, crawlee, scrapy, beautifulsoup/bs4, selectolax, puppeteer, cheerio.
+#
+# HOW TO RUN (inside YOUR project, not the skills repo)
+#   ./verify.sh                 # scan ./ for scraper source
+#   ./verify.sh --path src      # scan a subdirectory or a single file
+#   ./verify.sh --strict        # treat any warning as a failure (exit 1)
+#
+# EXIT CODES
+#   0  clean, or warnings only without --strict (also: nothing to check)
+#   1  an unbounded scrape (no pacing) or a circumvention call, or --strict + warn
+#   2  bad usage
+#
+# Runs on stock macOS bash 3.2 — no mapfile, no associative arrays.
+
+set -euo pipefail
+
+if [ -t 1 ]; then
+  RED=$'\033[31m'; GREEN=$'\033[32m'; YELLOW=$'\033[33m'; NC=$'\033[0m'
+else
+  RED=''; GREEN=''; YELLOW=''; NC=''
+fi
+
+ok_count=0; skip_count=0; warn_count=0; fail_count=0
+ok()   { printf '%s[ ok ]%s %s\n' "$GREEN"  "$NC" "$*"; ok_count=$((ok_count + 1)); }
+skip() { printf '%s[skip]%s %s\n' "$YELLOW" "$NC" "$*"; skip_count=$((skip_count + 1)); }
+warn() { printf '%s[warn]%s %s\n' "$YELLOW" "$NC" "$*"; warn_count=$((warn_count + 1)); }
+fail() { printf '%s[fail]%s %s\n' "$RED"    "$NC" "$*"; fail_count=$((fail_count + 1)); }
+
+usage() { sed -n '2,46p' "$0" | sed 's/^# \{0,1\}//'; }
+
+SCAN_PATH="."
+STRICT=0
+while [ $# -gt 0 ]; do
+  case "$1" in
+    --path)    SCAN_PATH="${2:?--path needs a value}"; shift 2 ;;
+    --strict)  STRICT=1; shift ;;
+    -h|--help) usage; exit 0 ;;
+    *) printf '%sUnknown argument: %s%s\n\n' "$RED" "$1" "$NC"; usage; exit 2 ;;
+  esac
+done
+
+if [ ! -e "$SCAN_PATH" ]; then
+  printf '%sPath not found: %s%s\n' "$RED" "$SCAN_PATH" "$NC"; exit 2
+fi
+
+# --- collect candidate source files -----------------------------------------
+TMPDIR_V="$(mktemp -d 2>/dev/null || printf '/tmp/dscrape-verify.%s' "$$")"
+mkdir -p "$TMPDIR_V" 2>/dev/null || true
+cleanup() { rm -rf "$TMPDIR_V" 2>/dev/null || true; }
+trap cleanup EXIT
+ALL="$TMPDIR_V/all"
+TARGETS="$TMPDIR_V/targets"
+
+find "$SCAN_PATH" -type f \
+  \( -name '*.py' -o -name '*.js' -o -name '*.ts' -o -name '*.mjs' -o -name '*.cjs' \) \
+  2>/dev/null > "$ALL" || true
+
+if [ ! -s "$ALL" ]; then
+  skip "no source files (*.py *.js *.ts *.mjs *.cjs) under: $SCAN_PATH"
+  printf '\nok=%d skip=%d warn=%d fail=%d\n' "$ok_count" "$skip_count" "$warn_count" "$fail_count"
+  exit 0
+fi
+
+# Keep only files that look like scrapers (import/use an HTTP/scrape library).
+SCRAPE_LIB='requests|httpx|curl_cffi|playwright|crawlee|scrapy|bs4|beautifulsoup|selectolax|puppeteer|cheerio'
+: > "$TARGETS"
+while IFS= read -r f; do
+  [ -z "$f" ] && continue
+  if grep -iEq "$SCRAPE_LIB" "$f" 2>/dev/null; then
+    printf '%s\n' "$f" >> "$TARGETS"
+  fi
+done < "$ALL"
+
+if [ ! -s "$TARGETS" ]; then
+  skip "no scraper source detected (no requests/httpx/playwright/crawlee/scrapy/... usage)"
+  printf '\nok=%d skip=%d warn=%d fail=%d\n' "$ok_count" "$skip_count" "$warn_count" "$fail_count"
+  exit 0
+fi
+
+n_targets="$(wc -l < "$TARGETS" | tr -dc '0-9')"
+ok "scraper source detected in $n_targets file(s)"
+
+# Helper: does ANY target file match a (case-insensitive) ERE pattern?
+any_match() {
+  pattern="$1"
+  while IFS= read -r f; do
+    [ -z "$f" ] && continue
+    if grep -iEq "$pattern" "$f" 2>/dev/null; then return 0; fi
+  done < "$TARGETS"
+  return 1
+}
+
+# Helper: list target files that match (for FAIL detail).
+list_match() {
+  pattern="$1"
+  while IFS= read -r f; do
+    [ -z "$f" ] && continue
+    if grep -iEq "$pattern" "$f" 2>/dev/null; then printf '  %s\n' "$f"; fi
+  done < "$TARGETS"
+}
+
+# --- 1. compliance preflight -------------------------------------------------
+PREFLIGHT='robots\.txt|robotstxt_obey|robotfileparser|robotparser|can_fetch|ai\.txt|terms of service|tos preflight|compliance preflight'
+if any_match "$PREFLIGHT"; then
+  ok "compliance preflight present (robots.txt / ToS / robotparser reference)"
+else
+  warn "no robots.txt / ToS / compliance preflight found — add the legal gate before requests"
+fi
+
+# --- 2. per-host pacing (the hard one) ---------------------------------------
+PACING='download_delay|crawl[_-]?delay|autothrottle|rate[_-]?limit|ratelimit|max_concurrency|concurrent_requests|max_requests_per|throttle|time\.sleep|asyncio\.sleep|setTimeout|sleep\('
+if any_match "$PACING"; then
+  ok "per-host pacing / concurrency cap present (delay / throttle / concurrency)"
+else
+  fail "no rate limit, delay, or concurrency cap found — unbounded scrape will get blocked/banned:"
+  list_match "$SCRAPE_LIB"
+fi
+
+# --- 3. retry / backoff ------------------------------------------------------
+BACKOFF='backoff|retry|retries|retry-after|retry_after|max_request_retries|exponential|tenacity'
+if any_match "$BACKOFF"; then
+  ok "retry / backoff present"
+else
+  warn "no retry/backoff found — transient errors will drop records; honor 429/Retry-After"
+fi
+
+# --- 4. no circumvention (the other hard one) --------------------------------
+CIRCUMVENT='2captcha|twocaptcha|anti-?captcha|anticaptcha|capsolver|deathbycaptcha|captcha[_-]?solver|solve[_-]?captcha|bypass[_-]?(auth|login|captcha|block)'
+if any_match "$CIRCUMVENT"; then
+  fail "circumvention call detected (CAPTCHA solver / auth-or-block bypass) — this is the legal line you do not cross:"
+  list_match "$CIRCUMVENT"
+else
+  ok "no CAPTCHA-solver / auth-bypass call detected"
+fi
+
+printf '\nok=%d skip=%d warn=%d fail=%d\n' "$ok_count" "$skip_count" "$warn_count" "$fail_count"
+
+if [ "$fail_count" -gt 0 ]; then exit 1; fi
+if [ "$STRICT" -eq 1 ] && [ "$warn_count" -gt 0 ]; then exit 1; fi
+exit 0

package/skills/db-migrations/SKILL.md

@@ -0,0 +1,237 @@
+---
+name: db-migrations
+description: "Use when a schema change must ship without downtime — NOT NULL, rename, type change, or backfilling millions of live rows — for the expand-contract sequence and the lock/batching discipline that keeps each step from freezing prod. NOT lock internals or EXPLAIN (that is `postgresdb`), NOT drizzle-kit mechanics (that is `drizzle-orm`), NOT PITR (that is `backups`)."
+tags: [migrations, zero-downtime, expand-contract, schema, backfill, ddl]
+recommends: [postgresdb, drizzle-orm, backups, sql, planetscale]
+origin: risco
+---
+
+# Database migrations without downtime
+
+A production migration is a **sequence**, not a statement. The moment you treat `ALTER TABLE ... NOT NULL`
+as one atomic change, you have already lost: the lock blocks every writer, the deploy that depends on the
+new column races the deploy that still reads the old one, and a rollback means a second outage. Your job
+is to decompose one risky change into a chain of small steps where **every intermediate state is
+independently deployable** — the old app keeps working, the new app keeps working, and you can stop at any
+point without an outage.
+
+This skill is engine-neutral. It owns the *choreography* — what order to run things in and how to keep each
+DDL from freezing the table — across Postgres, MySQL, and serverless variants, whatever runner you use
+(Flyway, Alembic, golang-migrate, drizzle-kit, raw SQL).
+
+## Route elsewhere
+
+- Postgres lock internals, EXPLAIN/ANALYZE, indexing strategy, RLS, PgBouncer → `../postgresdb/SKILL.md`.
+- Writing schema and running drizzle-kit specifically → `../drizzle-orm/SKILL.md`.
+- PlanetScale deploy-request / branch cutover (Vitess online DDL) → `../planetscale/SKILL.md`.
+- Creating/verifying backups, restore drills, PITR → `../backups/SKILL.md` (a migration *requires* a backup; making it lives there).
+- Pure query logic — joins, windows, CTEs → `../sql/SKILL.md` (backfill batching is here; query correctness is there).
+
+## Step 1 — Is this change online-safe in one step?
+
+Most outages come from running a multi-step change as a single statement. Decide first:
+
+| Change | One step? | Why |
+|---|---|---|
+| Add nullable column, no volatile default | Yes | Metadata-only on PG 11+/modern MySQL; no rewrite, no long lock. |
+| Add column with a **constant** default | Yes (PG 11+) | Stored as a catalog default, no table rewrite. |
+| Add column with a **volatile/expression** default | No | Rewrites every row under lock → expand-contract. |
+| Add `NOT NULL` to existing/new column | No | Full-table validation scan under lock → add-nullable → backfill → validate. |
+| Rename column / table | No | App reads the old name; needs dual-name window → expand-contract. |
+| Change column type (non-trivial) | No | Rewrites rows, may invalidate the running app → new column + backfill. |
+| Split / merge columns | No | Two-sided data move → expand-contract. |
+| Drop column / table | No | Running app may still reference it → contract only after a cooling period. |
+| Add foreign key | No | `ADD CONSTRAINT` validates all rows under lock → add `NOT VALID` then `VALIDATE`. |
+| Create index | No, but cheap | Plain `CREATE INDEX` locks writes → use `CONCURRENTLY`. |
+
+If the row says **No**, you are doing expand-contract. Go to Step 2.
+
+## Step 2 — The expand-contract loop
+
+Three phases, each tied to a **distinct deploy**. Worked example: rename `users.email` → `users.email_address`.
+
+**Expand** — add the new structure, start dual-writing, leave the old one fully working.
+
+```sql
+-- migration 1 (deploy A): additive only, old app untouched
+SET lock_timeout = '1s';
+ALTER TABLE users ADD COLUMN email_address text;  -- nullable, no default → metadata-only
+```
+
+```text
+App deploy A: write BOTH columns on every insert/update; keep READING email.
+```
+
+**Backfill** — copy historical data in small background batches (Step 4), then verify counts/checksums.
+
+**Switch reads** — once backfilled and verified, deploy the app that reads `email_address`.
+
+```text
+App deploy B: read email_address; still dual-write both.  ← cooling period starts here
+```
+
+**Contract** — only after a cooling period (hours to days) where the new app has run clean and you can
+still roll back the app without touching schema, stop writing the old column and drop it.
+
+```sql
+-- migration 2 (deploy C): runs days later, after cooling period
+SET lock_timeout = '1s';
+ALTER TABLE users DROP COLUMN email;
+```
+
+The cooling period is the point. It is what lets you roll back app deploy B to app deploy A with **zero
+schema change** — because the old column still exists and is still being written. Contracting early throws
+that safety away.
+
+## Step 3 — The N/N-1 rule
+
+At every moment, two app versions can be live: the one running and the one rolling out. **The schema must
+be compatible with both, and each app must tolerate the other's schema.** That is the entire reason
+dual-write exists — it makes the schema a superset that satisfies app N and app N+1 simultaneously.
+
+```text
+schema:  +email_address ......................... -email
+apps:    [N reads email] [N+1 reads email_address] [N+2 ignores email]
+              ^ both alive during rollout ^         ^ safe to drop now ^
+```
+
+If you ever have a single instant where one deployed app version cannot run against the current schema,
+your migration is not zero-downtime — it is a timed outage.
+
+## Step 4 — Make each DDL statement safe
+
+**Always set a short `lock_timeout` first.** A DDL waiting on a lock builds a *queue* — every query behind
+it stalls too, and one slow ALTER freezes the whole table. A short timeout makes the migration fail fast so
+traffic keeps flowing; you retry instead of taking the site down.
+
+```sql
+-- Bad: unbounded wait; if a lock is held, this queues every reader/writer behind it
+ALTER TABLE orders ADD COLUMN region text;
+
+-- Good: fail fast, retry
+SET lock_timeout = '1s';
+ALTER TABLE orders ADD COLUMN region text;
+```
+
+**Create indexes `CONCURRENTLY`, outside a transaction.** Plain `CREATE INDEX` takes a SHARE lock and
+blocks all writes for its whole duration — that is the "our deploy deadlocks when we add an index" symptom.
+`CONCURRENTLY` takes SHARE UPDATE EXCLUSIVE and does not block writes, but **cannot run inside a transaction
+block**. Runners (Flyway, Alembic, Django, Rails) wrap each migration in a transaction by default, so you
+must opt that step out — see `references/tools-and-runners.md`.
+
+```sql
+-- Bad: locks out every writer for the full build
+CREATE INDEX idx_orders_region ON orders (region);
+
+-- Good: non-blocking; this statement must NOT be inside BEGIN/COMMIT
+CREATE INDEX CONCURRENTLY idx_orders_region ON orders (region);
+```
+
+> A failed `CONCURRENTLY` build leaves an INVALID index behind. Drop it (`DROP INDEX CONCURRENTLY`) before
+> retrying. For PG-specific lock-level detail, see `../postgresdb/SKILL.md`.
+
+**Add NOT NULL in four steps, never in one.** A single `ALTER ... SET NOT NULL` (or `ADD COLUMN ... NOT
+NULL DEFAULT <volatile>`) scans/rewrites the whole table under lock.
+
+```sql
+-- Bad: full-table validation/rewrite under an exclusive lock
+ALTER TABLE users ADD COLUMN status text NOT NULL DEFAULT 'active';  -- on a 40M-row table = outage
+
+-- Good: add nullable, backfill in batches (Step 5), then add a constraint without a blocking scan
+SET lock_timeout = '1s';
+ALTER TABLE users ADD COLUMN status text;                              -- 1. nullable
+-- 2. backfill in batches (Step 5), app dual-writes the new column
+ALTER TABLE users ADD CONSTRAINT users_status_nn
+  CHECK (status IS NOT NULL) NOT VALID;                                -- 3. instant, no scan
+ALTER TABLE users VALIDATE CONSTRAINT users_status_nn;                 -- 4. scans WITHOUT blocking writes
+-- PG 12+: a validated CHECK (col IS NOT NULL) lets SET NOT NULL skip its own scan
+ALTER TABLE users ALTER COLUMN status SET NOT NULL;
+```
+
+Same `NOT VALID` → `VALIDATE` pattern for adding a foreign key. The full per-change checklists live in
+`references/expand-contract-playbook.md`.
+
+## Step 5 — Backfill at scale
+
+**Never one giant `UPDATE`.** A single statement over millions of rows holds locks, bloats WAL/undo, and
+pins replicas behind a wall of lag. Loop bounded chunks, commit each, throttle on lag, then verify.
+
+```sql
+-- Bad: one statement locks rows, balloons WAL, spikes replica lag for minutes
+UPDATE users SET email_address = email WHERE email_address IS NULL;
+```
+
+```sql
+-- Good: bounded PK-range batches, committed individually, throttled between batches
+-- pseudo-loop (driven by your runner/script): a few thousand rows per batch
+UPDATE users
+SET email_address = email
+WHERE id BETWEEN :lo AND :hi
+  AND email_address IS NULL;
+-- commit; check replica lag; sleep if lag/error-rate is high; advance :lo/:hi; repeat
+```
+
+Watch replication lag and the app error rate between batches; back off when either climbs. **Verify before
+contracting** — row counts and a checksum of old-vs-new must match. For very large sets, snapshot + CDC
+instead of an online loop. Postgres and MySQL backfill scripts, the lag-throttle, and checksum queries are
+in `references/backfill-and-batching.md`.
+
+## Step 6 — Rollback reality
+
+**Forward-only is the prevailing 2025-2026 production stance.** Reliable `down()` migrations are rarely
+worth the effort — whether a rollback is even safe depends on the migration type and how much time has
+passed (data has changed under you). Treat applied migrations as immutable and **fix forward**.
+
+Your real safety net is not a `down()` script — it is the expand-contract structure itself: because the old
+column/table still exists during the cooling period, "rolling back" is just redeploying the previous app.
+A reversible down-migration is only worth writing when the change is **additive-only and pre-traffic**
+(e.g. a brand-new table no app version reads yet). For anything that touched live data, a down-migration is
+a trap that looks like a seatbelt.
+
+## Step 7 — Pick a runner / OSC tool
+
+Versioned runners replay ordered scripts; declarative tools diff your target schema. Pick by stack:
+
+| Stack / need | Tool (2026) | Note |
+|---|---|---|
+| JVM, SQL-first, 50+ DBs | Flyway 12.6.x | Version-based; Redgate dropped the Teams tier in 2025. |
+| Max format flexibility | Liquibase | SQL/XML/YAML/JSON changelogs; most verbose. |
+| Python / SQLAlchemy | Alembic 1.18.x | Requires Python ≥3.10; plugin system since 1.18.0. |
+| Go | golang-migrate | The Go standard. |
+| TypeScript / Drizzle | drizzle-kit | Mechanics in `../drizzle-orm/SKILL.md`; safety sequence here. |
+| Schema-as-code, planned diffs | Atlas | Declarative — you give the target, it plans the diff. |
+| Big MySQL ALTER, native online DDL can't keep it online | gh-ost or pt-osc | See below. |
+
+**gh-ost vs pt-osc** for a large MySQL ALTER: **gh-ost** is triggerless — it copies in chunks and replays
+the binlog, throttles on replica lag, and gives easier pause/cutover control; prefer it on a very busy
+master. **pt-osc** is trigger-based — strictly consistent, but the triggers add write overhead and can fail
+to acquire the metadata lock under highly concurrent or long-transaction load. Decision detail and
+per-runner transaction opt-out in `references/tools-and-runners.md`.
+
+## Step 8 — Gate migrations in CI
+
+Add a static linter for migration SQL **before merge**, so a bare `CREATE INDEX` or a one-shot `NOT NULL`
+never reaches review. **Squawk** is the standard for Postgres migration SQL — it flags adding NOT NULL
+columns, non-concurrent index creation, blocking constraint adds, and column drops.
+
+```yaml
+# CI step: fail the PR on dangerous migration DDL
+- name: Lint migrations
+  run: npx squawk@latest migrations/*.sql
+```
+
+Config snippet and rule notes in `references/tools-and-runners.md`. `scripts/verify.sh` in this skill runs
+the same static checks against the example migrations shipped in `references/`.
+
+## Anti-patterns
+
+| Anti-pattern | Why it hurts | Do instead |
+|---|---|---|
+| `ALTER TABLE ... ADD COLUMN ... NOT NULL DEFAULT <expr>` on a big table | Full-table rewrite under lock = outage | add nullable → backfill → `NOT VALID` → `VALIDATE` (Step 4) |
+| Plain `CREATE INDEX` in prod | SHARE lock blocks all writes for the build → deploy "deadlock" | `CREATE INDEX CONCURRENTLY`, outside a transaction |
+| DDL with no `lock_timeout` | Builds a lock queue; one slow ALTER freezes the table | `SET lock_timeout='1s'` first, fail fast, retry |
+| One giant backfill `UPDATE` | Locks rows, bloats WAL, spikes replica lag | bounded batches, commit each, throttle on lag (Step 5) |
+| Renaming a column the running app still reads | Instant errors for app version N | dual-name window: add new, dual-write, switch reads, then drop |
+| Contracting before the cooling period | Kills your no-schema-change rollback path | wait until the new app has run clean; then drop |
+| Relying on `down()` in prod | Data already changed; rollback is unsafe or wrong | forward-only; expand-contract IS the rollback (Step 6) |
+| Coupling app + schema in one deploy | Breaks N/N-1; an instant where some app can't run | separate deploys; schema is a superset of both (Step 3) |

package/skills/db-migrations/evals/README.md

@@ -0,0 +1,10 @@
+# Evals — db-migrations
+
+`cases.yaml` specifies routing and capability expectations, not live database runs. `should_trigger` /
+`should_not_trigger` are prompt-routing checks: feed each prompt to the router and confirm db-migrations
+fires (or that the named sibling — postgresdb, drizzle-orm, backups, sql, planetscale — wins instead).
+The `capability` case is a rubric: ask the model to plan the column rename, then score the response
+against the `must_include` bullets (each phase present, in order, with lock_timeout, batched/throttled
+backfill, checksum verify, N/N-1, cooling period, forward-only). There is no database connection here;
+the DDL-shape and self-consistency check lives in `../scripts/verify.sh`. Run these through the repo's
+standard eval harness.

package/skills/db-migrations/evals/cases.yaml

@@ -0,0 +1,46 @@
+skill: db-migrations
+
+should_trigger:
+  - prompt: "We need to add a NOT NULL column to a 40M-row users table without taking the app down. How?"
+    why: Core expand-contract — add-nullable, backfill, NOT VALID/VALIDATE, on a live large table.
+  - prompt: "Rename a heavily-written column on a live Postgres table without downtime."
+    why: Dual-name window with dual-write + batched backfill; the canonical sequencing case.
+  - prompt: "Our deploy keeps deadlocking every time the migration adds an index. What are we doing wrong?"
+    why: Non-obvious — symptom names a deadlock, not a migration; the fix is CONCURRENTLY + lock_timeout.
+  - prompt: "How do we backfill 80M rows into a new column without blowing up replication lag?"
+    why: Batched backfill with lag throttle and verify-before-contract; not query logic.
+  - prompt: "Necesito hacer una migración sin downtime para cambiar el tipo de una columna en producción."
+    why: Spanish phrasing for a type change on a live table — new column + backfill expand-contract.
+  - prompt: "Big ALTER on a busy MySQL table — should we use gh-ost or pt-osc?"
+    why: Non-obvious OSC-tool choice; engine-neutral choreography layer owns this decision.
+
+should_not_trigger:
+  - prompt: "This query is slow — can you read the EXPLAIN ANALYZE plan and tell me which index to add?"
+    route_to: postgresdb
+    why: Query plan reading and Postgres indexing strategy are postgresdb internals, not migration sequencing.
+  - prompt: "How do I generate and run a migration with drizzle-kit?"
+    route_to: drizzle-orm
+    why: That is the runner's CLI mechanics; the safety sequence is here but the tool usage lives in drizzle-orm.
+  - prompt: "Set up a PITR restore drill and verify our nightly backups actually restore."
+    route_to: backups
+    why: Creating and verifying backups/restores is backups; a migration only requires one as a precondition.
+  - prompt: "Write a window-function query to rank orders per customer by date."
+    route_to: sql
+    why: Pure query logic (window functions) is sql, not schema-change choreography.
+  - prompt: "Cut over a PlanetScale branch and merge the deploy request."
+    route_to: planetscale
+    why: Vitess online DDL and deploy-request cutover mechanics belong to planetscale.
+
+capability:
+  - scenario: >
+      Plan, end to end, a zero-downtime rename of a heavily-written column users.email →
+      users.email_address on Postgres, across app deploys and migrations.
+    must_include:
+      - "Expand: add email_address as a NULLABLE column, metadata-only, with lock_timeout set first"
+      - "App deploy that dual-writes both email and email_address while still reading email"
+      - "Batched backfill (bounded PK-range chunks, commit per batch) with replica-lag throttle"
+      - "Verify counts and old-vs-new checksum before contracting"
+      - "N/N-1 note: schema must satisfy both the running app and the next app at every step"
+      - "Switch reads to email_address in a separate deploy, then a cooling period"
+      - "Contract: DROP COLUMN email as a separate migration AFTER the cooling period"
+      - "Forward-only stance: rollback is redeploying the prior app, not a down() script"