sanook-cli 0.4.0 → 0.5.1

package/skills/deliver-webhooks/SKILL.md

@@ -0,0 +1,116 @@
+---
+name: deliver-webhooks
+description: Builds the producer side of webhooks — you dispatch signed events to customers' HTTPS endpoints. Sign every payload with HMAC-SHA256 over "{timestamp}.{raw_body}" in a versioned signature header with per-endpoint secrets and rotation overlap; deliver at-least-once with exponential backoff + jitter over hours, then dead-letter with manual replay; send thin events (id, type, ts, minimal data) so consumers re-fetch the resource; isolate delivery per endpoint so one broken target can't stall everyone; ship a stable event id + sequence number so consumers can dedup and not assume order; verify endpoints on registration; and lock down the target URL against SSRF (HTTPS-only, block internal/link-local/metadata IPs, re-resolve on each send). Use when your service must reliably and verifiably push events out to third-party subscribers.
+when_to_use: You are the SENDER pushing events to customers' webhook URLs (the Stripe/GitHub side) — building event dispatch, payload signing, the retry+DLQ schedule, endpoint registration, or a delivery-history/replay UI. Distinct from ingest-webhook-secure (the RECEIVER side — verifying inbound signatures and safely processing webhooks you consume) and message-queue-jobs (the general internal job system used here as the delivery substrate; this skill adds the webhook-specific signing, replay, SSRF, and consumer-ergonomics layer on top).
+---
+
+## When to Use
+
+Reach for this skill when **your service emits events that third parties subscribe to** and you must deliver them verifiably and reliably:
+
+- "Let customers register a webhook URL and we POST events to it when X happens"
+- "How do we sign payloads so receivers can verify the request really came from us?"
+- "One customer's endpoint is down/slow and it's backing up deliveries for everyone"
+- "A delivery failed — retry it with backoff, then dead-letter, with a replay button in the dashboard"
+- "Rotate a customer's signing secret without dropping any deliveries"
+- "Add a delivery-history / attempts log to the customer dashboard"
+- "Someone registered `http://169.254.169.254/...` as their webhook URL" (SSRF)
+
+NOT this skill:
+- *Receiving* and verifying inbound webhooks from a provider (Stripe→you) → ingest-webhook-secure (that's the mirror image: you verify their signature; here you produce yours)
+- The underlying job queue / worker / DLQ plumbing (SQS/Kafka/BullMQ/Celery) → message-queue-jobs — used here as the substrate; this skill is the webhook policy on top
+- The retry/backoff/jitter/circuit-breaker math for outbound calls → resilience-timeouts-retries (the primitive this skill's delivery schedule is built on)
+- Per-endpoint request-rate caps / token bucket → rate-limiting (this skill references it for per-target throttling, doesn't reimplement it)
+- Making the *consumer* safe to re-process a redelivered event → idempotency-keys (your job is to send a stable id + sequence so they *can*)
+- Where the per-endpoint signing secret is stored/encrypted at rest → secrets-management
+- Delivery metrics/traces/dashboards plumbing → observability-instrument
+
+## Steps
+
+1. **Sign every payload — HMAC-SHA256 over the exact bytes you send, with a per-endpoint secret.** Compute the signature over `"{timestamp}.{raw_body}"` (the same bytes on the wire — serialize ONCE, sign that buffer, send that buffer; never re-serialize between signing and sending or receivers' verification breaks). Put it in a versioned header so you can add schemes later without breaking verifiers:
+
+   ```
+   X-Webhook-Id: evt_01HZ...            # stable unique event id (also a dedup key for the consumer)
+   X-Webhook-Timestamp: 1718409600      # unix seconds; part of the signed preimage
+   X-Webhook-Signature: t=1718409600,v1=5257a8...   # v1 = hex HMAC-SHA256(secret, "{t}.{body}")
+   ```
+   ```python
+   import hmac, hashlib, json
+   raw = json.dumps(payload, separators=(",", ":"), sort_keys=True).encode()  # serialize ONCE
+   t = str(int(time.time()))
+   sigs = [f"v1={hmac.new(s, f'{t}.'.encode()+raw, hashlib.sha256).hexdigest()}"
+           for s in active_secrets_for(endpoint)]   # one secret per endpoint; >1 during rotation
+   headers = {"X-Webhook-Id": event_id, "X-Webhook-Timestamp": t,
+              "X-Webhook-Signature": "t=" + t + "," + ",".join(sigs)}
+   ```
+   One secret **per endpoint** (never a global secret — a leak then compromises every customer). Document the verify recipe for receivers and point them at ingest-webhook-secure.
+
+2. **Support secret rotation with an overlap window — send multiple signatures.** Rotation = generate a new secret, mark both old+new *active*, and sign with **both** during the overlap (`v1=<old>,v1=<new>`). The receiver accepts a request if *any* signature matches, so they can swap secrets at their leisure. After the documented overlap (e.g. 24–72 h, or when the customer confirms), retire the old secret. Without overlap, rotation drops every in-flight delivery. Store secrets via secrets-management; show "rotate" + "reveal once" in the dashboard.
+
+3. **Include a signed timestamp + document a tolerance window so receivers can reject replays.** The `t` you sign lets a receiver drop a captured-and-replayed request that's older than their tolerance (recommend **±300 s**). You can't enforce it — but you must (a) put `t` *inside* the signed preimage (not just a loose header), (b) keep your senders' clocks NTP-synced so legit deliveries land inside the window, and (c) document the recommended tolerance so receivers implement it. Drift on your side = false replay rejections at every customer.
+
+4. **Deliver at-least-once with exponential backoff + jitter over hours/days, then dead-letter.** Treat **any non-2xx, timeout, or connection error as retryable**; 2xx (any) = delivered, stop. Build the schedule on resilience-timeouts-retries (full jitter, per-attempt timeout ~10 s). Give up after N attempts (e.g. 8–15) spread across a long horizon, then move to a **dead-letter store** with a manual **replay** button.
+
+   | Attempt | Delay (base, +jitter) | Cumulative |
+   |---|---|---|
+   | 1 | immediate | 0 |
+   | 2 | ~30 s | ~30 s |
+   | 3 | ~2 m | ~3 m |
+   | 4 | ~10 m | ~13 m |
+   | 5 | ~1 h | ~1 h |
+   | 6 | ~3 h | ~4 h |
+   | 7–N | ~6 h, capped | up to ~1–3 days |
+
+   After exhaustion → DLQ row with last status/error; surface it in the dashboard with a one-click "replay" that re-enqueues the *same event* (same id + sequence) so consumer dedup still works. Auto-disable an endpoint that's failed for days and email the owner.
+
+5. **Send a STABLE unique event id and a per-endpoint SEQUENCE number — you WILL re-deliver and you do NOT guarantee order.** The `event_id` is generated once at event creation and is identical across every retry of that event (it's the consumer's dedup key → idempotency-keys). Add a monotonic `sequence` (per endpoint or per resource) **and** the `timestamp` so consumers can detect/repair reordering. Explicitly document: *delivery is at-least-once and unordered — retries and parallel sends mean `updated` can arrive before `created`; dedup on `event_id`, order on `sequence`/`timestamp`, never on arrival order.* Don't pretend ordering you can't deliver.
+
+6. **Send THIN events; let the consumer fetch the full resource.** Payload = `{ id, type, timestamp, sequence, data: { id, <a few key fields> } }` — enough to route and decide, not the whole object. Then the consumer GETs `/v1/orders/{id}` from your API for current truth. This avoids (a) **stale payloads** (the resource changed between enqueue and delivery), (b) **oversized bodies** that blow timeouts, and (c) **leaking** fields the subscriber shouldn't see / that bloat your audit logs. For events that are inherently terminal facts (`invoice.finalized` snapshot), a fuller payload is fine — but default thin.
+
+7. **Version the event schema and keep it stable.** Give every event a `type` (`order.created`) and an explicit schema version (`"api_version": "2026-06-01"` on the event, or `v1` in the signature header). **Add fields, never repurpose/remove** within a version; breaking changes = a new version that customers opt into. Publish a typed catalog of event types + JSON schema. Stripe-style dated versions or a coarse `v1/v2` both work — pick one and document it.
+
+8. **Verify the endpoint on registration before sending real traffic.** When a customer adds a URL, prove they control it and it works: send a **test/`ping` event** (or a challenge-response where they echo a token) and require a **2xx within timeout** before marking the endpoint active. This blocks typos, dead URLs, and registering *someone else's* endpoint to flood it. Re-verify on URL change. Keep the endpoint in `pending` until the test succeeds.
+
+9. **Lock the target down against SSRF — your dispatcher is a server-side request to a customer-controlled URL.** This is the highest-severity bug in any webhook sender. On registration AND on **every send** (DNS rebinding defeats register-time-only checks):
+
+   - **HTTPS only.** Reject `http://`, `file://`, `gopher://`, etc.
+   - **Resolve the hostname yourself, then block** loopback/private/link-local/metadata ranges: `127.0.0.0/8`, `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `169.254.0.0/16` (incl. cloud metadata `169.254.169.254`), `::1`, `fc00::/7`, `fe80::/10`, `0.0.0.0`. Block by *resolved IP*, not by string-matching the hostname.
+   - **Pin the connection to the IP you validated** (connect to the checked IP / re-resolve and re-check just before connect) so a TOCTOU re-resolution can't swap in an internal IP between check and connect.
+   - **Disable redirect following** (or re-validate every hop) — a 302 to `http://169.254.169.254` bypasses a register-time check.
+   - Egress from a **locked-down network** (no access to internal services / metadata endpoint) as defense in depth. Cap response body read and per-attempt timeout. See remediate-web-vulnerabilities for the SSRF class.
+
+10. **Isolate delivery per endpoint/customer so one bad target can't stall the rest.** Run delivery on message-queue-jobs, but partition: a **per-endpoint queue/lane** with **bounded concurrency**, so a customer whose endpoint hangs/5xx's only backs up *their* lane. Add a **per-endpoint rate cap** (→ rate-limiting) to respect receivers' limits, and a **circuit breaker** (→ resilience-timeouts-retries) that fast-fails (straight to the retry schedule) while an endpoint is consistently down instead of burning worker time on it. Never deliver all customers off one shared FIFO — head-of-line blocking will take down delivery for everyone the moment one endpoint goes slow.
+
+11. **Observability — per-attempt logs + a customer-facing delivery history and replay UI.** Log **every attempt** (not just final): event id, endpoint, attempt #, response status, latency, error, next-retry-at. Emit metrics per endpoint: **success rate, attempt count, p50/p95 latency, dead-letter count** (→ observability-instrument). Expose to the customer a **delivery history** showing each event, its attempts, response codes/bodies (truncated), and a **manual replay** button — this is what turns "your webhooks are broken" support tickets into self-service. Alert the owner when an endpoint's success rate craters or it gets auto-disabled.
+
+## Common Errors
+
+- **Re-serializing the body between signing and sending.** Sign a buffer, then a middleware/pretty-printer/key-reorder changes the bytes → every receiver's HMAC fails. Serialize ONCE, sign and send the *same* bytes.
+- **One global signing secret for all endpoints.** A single leak compromises every customer and forces a flag-day rotation. Use one secret per endpoint.
+- **Rotation with no overlap.** Swap the secret atomically → every in-flight and newly-signed delivery fails verification at the receiver. Send both old+new signatures during the overlap window, retire old after.
+- **Timestamp not inside the signature (or unsynced clocks).** A loose `X-Timestamp` header an attacker can edit gives no replay protection; NTP-unsynced senders make legit deliveries fall outside receivers' tolerance. Sign `"{t}.{body}"`; keep clocks synced.
+- **Treating 2xx-but-slow as success without a per-attempt timeout.** A hanging endpoint pins a worker forever. Bound each attempt (~10 s) and count a timeout as a retryable failure.
+- **No dead-letter / no replay.** After N retries the event vanishes silently and the customer never knows. DLQ + a manual replay that re-sends the same event id.
+- **Assuming/promising ordered delivery.** Retries + parallel lanes reorder events; consumers that apply in arrival order corrupt state. Send `sequence` + `timestamp`, document "unordered, at-least-once," and tell consumers to dedup + order on those fields.
+- **Fat payloads with the full resource.** Stale by delivery time, oversized, and leak fields. Send thin + an `id`; let the consumer re-fetch.
+- **SSRF: validating the URL string but not the resolved IP, or only at registration.** `http://internal.svc`, a hostname that resolves to `169.254.169.254`, or DNS-rebinding after registration all hit internal services / cloud metadata. Resolve + block private/link-local/metadata ranges on **every** send, pin to the validated IP, disable redirects.
+- **Following redirects blindly.** A `302 → http://169.254.169.254/latest/meta-data/` turns a clean external URL into an SSRF. Don't follow, or re-validate each hop.
+- **Shared global delivery queue.** One slow endpoint head-of-line-blocks everyone. Partition per endpoint with bounded concurrency + a breaker.
+- **No endpoint verification on registration.** Typo'd/dead/hijacked URLs accepted; you send real events into the void or at a victim. Require a challenge / test event returning 2xx before activating.
+- **Per-attempt logs missing.** Only logging final outcome makes "why did delivery 5 fail at 14:03" undebuggable. Log every attempt with status/latency/error and expose it to the customer.
+
+## Verify
+
+1. **Signature round-trips:** an independent verifier (the ingest-webhook-secure recipe) recomputes `HMAC("{t}.{body}")` over the received raw bytes and matches `v1` exactly; flipping one body byte fails verification.
+2. **Per-endpoint secrets:** two endpoints get different signatures for the same event; a secret leaked from one does not verify the other.
+3. **Rotation overlap:** during rotation the request carries two `v1` signatures; a receiver holding the old secret AND one holding the new secret both verify; after overlap, only the new verifies.
+4. **Replay window honored:** delivered `t` is inside ±tolerance of real time (clock-sync check); a receiver rejecting `t` older than tolerance still accepts your live deliveries.
+5. **Retry schedule + DLQ:** point an event at an endpoint that returns 500 → it retries with growing, jittered delays, stops after N attempts, lands in the DLQ; the replay button re-sends the **same** event id and a now-healthy endpoint accepts it once.
+6. **Stable id + sequence:** every retry of one event carries the identical `event_id`; `sequence` is monotonic per endpoint; deliver two events out of order and confirm the consumer can reorder via `sequence`/`timestamp`.
+7. **Thin payload:** body contains id/type/timestamp/sequence + minimal data only; the documented re-fetch returns current truth even after the resource changed post-enqueue.
+8. **Endpoint isolation:** make endpoint A hang/timeout; endpoint B keeps receiving on time (assert B's delivery latency is unaffected) — proves no head-of-line blocking.
+9. **SSRF blocked:** registering/sending to `http://x` (non-HTTPS), `https://127.0.0.1`, `https://169.254.169.254`, a `10.x`/`::1` address, or a hostname that resolves into a private range is rejected — at registration AND on send (test DNS-rebinding: hostname resolves public at register, private at send → still blocked); a 302 to a metadata IP is not followed.
+10. **Registration verification:** a URL that never returns 2xx to the test event stays `pending`/inactive and receives no real events.
+11. **Observability:** every attempt produces a log row (event id, endpoint, attempt #, status, latency); the customer dashboard lists deliveries + attempts and the replay button works.
+
+Done = each event is HMAC-signed per-endpoint over the exact bytes with a signed timestamp and rotation overlap, delivery is at-least-once with jittered backoff → DLQ → manual replay, events are thin and carry a stable id + sequence so consumers dedup and reorder, the target URL is HTTPS-only and SSRF-blocked (private/link-local/metadata ranges, re-checked on every send), endpoints are verified before activation, delivery is isolated per endpoint, and every attempt is logged and visible to the customer with a working replay — all proven by checks 1–11.

package/skills/design-api-pagination/SKILL.md

@@ -0,0 +1,144 @@
+---
+name: design-api-pagination
+description: Designs paginated list endpoints that stay correct and fast under concurrent writes — cursor/keyset pagination over a stable total ordering with a unique tie-break key (e.g. ORDER BY created_at DESC, id DESC and WHERE (created_at,id) < (?,?)), opaque base64url-encoded cursors that bind sort+filter so they can't be tampered or reused across queries, a sane page_size default (20-50) and hard cap (100), and the {data, next_cursor, has_more} envelope (fetch limit+1 to compute has_more without a COUNT) — instead of OFFSET/LIMIT, which gets O(n) slow on deep pages and skips/duplicates rows when items are inserted or deleted mid-scan; covers REST and GraphQL Relay connections (edges/node/cursor + pageInfo.hasNextPage/endCursor), forward+backward paging, and why total counts are expensive and usually optional.
+when_to_use: Building or fixing a list/feed/search endpoint that returns many rows and needs paging, an infinite-scroll or "load more" API, a stable cursor under live inserts/deletes, or migrating a slow OFFSET endpoint to keyset; or implementing a GraphQL Relay connection. Distinct from api-design-review (reviews the whole API surface/REST conventions; this owns the pagination mechanics specifically) and optimize-sql-query (builds the covering composite index that makes the keyset WHERE/ORDER BY fast; this decides the cursor/ordering contract that index must serve).
+---
+
+## When to Use
+
+Reach for this skill when an endpoint returns a list that's too big for one response and must page through it correctly:
+
+- "Add pagination to this list/feed/search endpoint" / "support infinite scroll / load-more"
+- "Our `?page=500` query takes 8 seconds — deep OFFSET is killing us"
+- "Users see duplicate or missing rows while scrolling a live feed" (rows inserted/deleted mid-scan)
+- "Design the cursor — should it be opaque? what goes in it?"
+- "Implement a GraphQL Relay connection (edges/pageInfo/cursors)"
+- "We need stable ordering with a tie-break so pages don't shuffle"
+- "Do we have to return a total count?" (usually no — it's the expensive part)
+
+NOT this skill:
+- Reviewing the whole REST/HTTP API surface — resource naming, status codes, versioning, error shape → api-design-review (this skill is only the pagination contract within it)
+- Defining the serialized field types / GraphQL schema contract in general → rest-graphql-contract (this skill specifies the connection/cursor shape it slots into)
+- Building the composite/covering index that makes the keyset `WHERE (a,b) < (?,?)` fast, EXPLAIN-tuning the scan → optimize-sql-query (this skill defines the ordering the index must support)
+- Caching list responses / CDN / ETag for pages → caching-strategy
+- Rate-limiting how many pages a client can pull → rate-limiting
+- Throttling/queuing expensive list jobs → message-queue-jobs
+- Designing the underlying table/keys → design-relational-schema (this skill consumes the unique key it needs as a tie-break)
+
+## Steps
+
+1. **Default to keyset (cursor) pagination; reach for OFFSET only for small, static, jump-to-page-N admin tables.** The two models:
+
+   | | Offset/limit | Keyset/cursor |
+   |---|---|---|
+   | Query | `ORDER BY ... LIMIT 20 OFFSET 980` | `WHERE (sort_key,id) < (?,?) ORDER BY ... LIMIT 20` |
+   | Deep-page cost | **O(offset)** — DB scans + discards all skipped rows | **O(1)** w/ index — seeks straight to the cursor |
+   | Concurrent insert/delete | **skips or duplicates** rows (offset shifts under you) | stable — anchored to a value, not a position |
+   | Jump to page N | yes | no (sequential only) |
+   | Total pages | derivable (needs COUNT) | not directly |
+
+   Offset is fine for a 200-row config table behind admin; for any feed, search, timeline, or table that grows or is written concurrently, **keyset is the default**.
+
+2. **Pick a stable total ordering with a unique tie-break — this is the whole game.** The `ORDER BY` columns must be (a) the user-visible sort and (b) **made total** by appending a unique, immutable column (the PK) so no two rows compare equal. A non-unique sort (`ORDER BY created_at` alone) lets rows with the same timestamp straddle a page boundary → duplicates or skips.
+
+   ```sql
+   -- newest-first feed, made total by id tie-break
+   ORDER BY created_at DESC, id DESC
+   ```
+   The cursor encodes the **full** sort tuple of the last row returned: `(created_at, id)`. Use row-value comparison so it's one index seek:
+   ```sql
+   WHERE (created_at, id) < (:last_created_at, :last_id)   -- DESC page
+   ORDER BY created_at DESC, id DESC
+   LIMIT :page_size + 1;
+   ```
+   For ASC use `>`. For **mixed** directions (`created_at DESC, name ASC`) row-value syntax doesn't apply — expand the boolean predicate explicitly:
+   ```sql
+   WHERE created_at < :c
+      OR (created_at = :c AND name > :n)
+      OR (created_at = :c AND name = :n AND id > :id)
+   ```
+   Tie-break key must be **unique and never-updated** (PK, not a mutable slug). If the sort column itself is mutable (e.g. `updated_at`), a row can move pages — acceptable for "recently updated" feeds, surprising for "newest"; document it.
+
+3. **Make cursors opaque and self-describing — base64url-encode a small payload, never expose raw offsets/ids.** A cursor is a token the client echoes back verbatim; it is NOT a stable id and clients must not parse it. Encode the sort tuple plus enough to detect misuse:
+
+   ```json
+   { "k": [1718409600000, 84213], "d": "desc", "f": "a1b2c3" }
+   // k = sort-key tuple of last row, d = direction, f = hash of filter+sort params
+   ```
+   `base64url(JSON)` → `eyJrIjpb...`. Rules:
+   - **Opaque:** document it as "treat as opaque; do not construct or parse." Lets you change the internal format later without breaking clients.
+   - **Bind the query shape:** include `f` = a hash (or the canonical filter/sort) the cursor was created under. On the next request, **reject (400) if the client changed `filter`/`sort` but reused the cursor** — a cursor is only valid against the exact query that produced it.
+   - **Don't trust it for authz:** re-apply tenant/visibility filters on every page; never assume the cursor proves access. Tamper-resistance optional — sign (HMAC) only if a forged cursor could leak data past a filter; usually re-filtering server-side is enough.
+   - **Cross-page consistency:** a cursor's sort tuple is independent of position, so inserts/deletes between pages don't shift it — the core win over offset.
+
+4. **Set a page-size default and a hard cap; fetch `limit + 1` to compute `has_more` cheaply.** Never let the client ask for unbounded rows (DoS / memory blowup).
+
+   | Param | Value |
+   |---|---|
+   | default `page_size` | 20–50 |
+   | hard max | 100 (clamp, don't 400 — `min(requested, 100)`) |
+   | `limit + 1` trick | query `page_size + 1`; if you got the extra row, `has_more = true`, drop it from `data`, its predecessor's key is `next_cursor` |
+
+   This avoids a separate `COUNT(*)` just to know if there's a next page. Reject `page_size <= 0`.
+
+5. **Return the `{data, next_cursor, has_more}` envelope; make `next_cursor` null at the end.** Stable, minimal contract:
+   ```json
+   {
+     "data": [ /* page_size items */ ],
+     "next_cursor": "eyJrIjpbMTcxODQw...",  // null when has_more=false
+     "has_more": true
+   }
+   ```
+   - `next_cursor` is the cursor of the **last returned row** (after dropping the `+1` probe). Client passes it back as `?cursor=...`.
+   - When `has_more=false`, `next_cursor=null` and clients stop — don't make them request an empty page to discover the end.
+   - **Omit total by default.** A `total_count` forces a full `COUNT(*)` (often as slow as the data scan) and is meaningless under concurrent writes. Offer it only as an opt-in (`?include_total=true`), cache it, or return an estimate (`reltuples` / approximate count).
+
+6. **Support backward paging when the UI needs "previous" — flip comparison + order, then re-reverse.** For bidirectional cursors carry the direction in the token. To page backward from a cursor: flip `<`→`>` (and the `ORDER BY` direction), `LIMIT n+1`, then **reverse the returned slice in memory** so the client still gets ascending-by-display order. Track both `has_next` and `has_prev`. Relay's `pageInfo` (next step) formalizes this with `hasNextPage`/`hasPreviousPage` + `startCursor`/`endCursor`.
+
+7. **For GraphQL, follow the Relay Cursor Connections spec exactly — don't invent a connection shape.** REST and GraphQL share the same keyset engine; only the envelope differs. Relay structure:
+   ```graphql
+   type PostConnection {
+     edges: [PostEdge!]!
+     pageInfo: PageInfo!
+   }
+   type PostEdge { node: Post!  cursor: String! }   # per-row opaque cursor
+   type PageInfo {
+     hasNextPage: Boolean!  hasPreviousPage: Boolean!
+     startCursor: String    endCursor: String
+   }
+   # query args: first/after (forward), last/before (backward)
+   ```
+   - `first: 20, after: "<cursor>"` is forward; `last: 20, before: "<cursor>"` is backward. Each `edge.cursor` is the opaque keyset token for that node.
+   - `pageInfo.endCursor` ↔ REST `next_cursor`; `hasNextPage` ↔ `has_more`. `totalCount` is a **separate, optional** field — same COUNT cost caveat (step 5).
+   - Don't mix `first` with `last`, or `after` with `before`, in one request — reject it.
+
+8. **Hand the ordering to `optimize-sql-query` and make sure a composite index covers it.** Keyset is only O(1) if a B-tree index matches the `ORDER BY` columns **in order and direction**: `CREATE INDEX ON posts (created_at DESC, id DESC)` (plus any equality filter columns as a **leading prefix**: `(tenant_id, created_at DESC, id DESC)` for a per-tenant feed). Without it the DB sorts the whole table per page and you've gained nothing. Verify with `EXPLAIN` that it's an index range scan with no `Sort` node and `Rows Removed by Filter ≈ 0`.
+
+## Common Errors
+
+- **OFFSET for deep pages on a growing table.** `OFFSET 100000` scans and throws away 100k rows; latency grows linearly with page depth. Fix: keyset/cursor (step 1).
+- **Non-unique `ORDER BY` (no tie-break).** `ORDER BY created_at` with duplicate timestamps → rows straddle page boundaries, appear twice or vanish. Fix: append the unique PK to make ordering total (step 2).
+- **Exposing a raw offset/id/timestamp as the "cursor."** Clients build their own, you can't change the format, and they construct invalid ones. Fix: opaque base64url token, documented as un-parseable (step 3).
+- **Cursor not bound to the query.** Client keeps the cursor but switches `sort` or `filter` → garbage page or skipped rows. Fix: encode a filter/sort hash in the cursor and 400 on mismatch (step 3).
+- **No page-size cap.** `?page_size=1000000` OOMs the server. Fix: default 20–50, clamp to max 100 (step 4).
+- **Separate `COUNT(*)` on every page for `has_more`.** Doubles DB load. Fix: fetch `limit + 1` and check for the extra row (step 4).
+- **Mandatory `total_count`.** Forces a full count scan, and it's wrong under concurrent writes anyway. Fix: omit by default; opt-in / cached / estimated (step 5).
+- **Sorting on a mutable column without telling anyone.** Ordering by `updated_at` lets a row jump pages mid-scroll → silent dup/skip. Fix: prefer immutable sort (created_at/id); if mutable, document the behavior.
+- **Missing/mismatched index.** Keyset query without a composite index matching column order+direction → full sort per page, no speedup. Fix: index the exact `(filter…, sort…, id)` tuple, verify no `Sort` in EXPLAIN (step 8).
+- **Row-value comparison with mixed sort directions.** `(a,b) < (?,?)` is wrong when `a` and `b` sort opposite ways. Fix: expand to the explicit OR-chain predicate (step 2).
+- **GraphQL inventing its own `{items, nextPage}` instead of Relay connections.** Breaks Relay/Apollo client cache + tooling assumptions. Fix: follow edges/node/cursor + pageInfo (step 7).
+- **Off-by-one at the boundary.** Forgetting to drop the `+1` probe row leaks it into `data` and as the cursor. Fix: slice to `page_size`, derive `next_cursor` from the last *kept* row.
+
+## Verify
+
+1. **Deep page is fast:** request the millionth row's page; latency ≈ first page (constant), not linear. `EXPLAIN ANALYZE` shows an index range scan, no `Sort` node, near-zero rows filtered.
+2. **Stable under inserts:** start paging, insert/delete rows ahead of and behind the cursor mid-scan; assert no row appears twice and no existing-before-the-cursor row is skipped (the offset failure mode).
+3. **Tie-break holds:** seed many rows with identical sort values (same `created_at`); page through and assert every row appears exactly once across page boundaries.
+4. **Cursor is opaque + bound:** decode shows no client-meaningful offset; reusing a cursor with a changed `filter`/`sort` returns 400, not a corrupt page.
+5. **Page size enforced:** `page_size=1000000` returns ≤100; `page_size=0`/negative is rejected.
+6. **End-of-list is clean:** the last page returns `has_more=false` and `next_cursor=null`; clients never need an extra empty request to detect the end.
+7. **`has_more` without COUNT:** confirm the query plan fetches `limit+1` and runs no `COUNT(*)` unless `include_total` is explicitly set.
+8. **Bidirectional round-trip:** page forward N then backward N lands on the original rows in the original display order (slice was re-reversed correctly).
+9. **Relay conformance (GraphQL):** `first/after` and `last/before` work; `pageInfo.hasNextPage`/`endCursor` are correct; mixing `first` with `before` is rejected; `totalCount` is opt-in.
+
+Done = list endpoints use keyset cursors over a unique-tie-break total ordering, cursors are opaque base64url tokens bound to their query, page size is defaulted and hard-capped, `has_more` comes from `limit+1` (no mandatory COUNT), the `{data,next_cursor,has_more}` (or Relay connection) envelope is stable, a composite index backs the ordering, and the consistency/perf tests in checks 1–9 pass under concurrent writes.

package/skills/design-authorization-model/SKILL.md

@@ -0,0 +1,119 @@
+---
+name: design-authorization-model
+description: Designs an authorization model — RBAC/ABAC/ReBAC, multi-tenant isolation, resource ownership, and policy-as-code (OPA/Cedar/Oso) — keeping authZ decisions separate from authN identity in a centralized, testable policy layer enforced down to the data tier.
+when_to_use: A system needs roles/permissions, multi-tenant data isolation, or per-resource access rules beyond a logged-in check. Distinct from auth-jwt-session (who you are — tokens/sessions), security-review (audit), and rate-limiting (request volume).
+---
+
+## When to Use
+
+Reach for this skill when the question is **"is this caller allowed to do this to this resource?"** — not "who is this caller?":
+
+- "Add roles and permissions" / "only admins can delete, members can edit, viewers read"
+- "Tenants must not see each other's data" / "isolate orgs / workspaces"
+- "Owner can share a doc with specific users" (Google-Drive-style) → relationship graph
+- "Permissions depend on attributes" — department, resource status, time, region
+- "Stop scattering `if user.role == 'admin'` across 40 handlers — centralize it"
+- An IDOR/cross-tenant leak found in review (a user fetched another org's record by id)
+
+NOT this skill:
+- Issuing/verifying tokens, sessions, refresh rotation, OAuth/OIDC login → **auth-jwt-session** (authN establishes identity; this skill consumes that identity to make the access decision)
+- Auditing existing code for access-control holes by severity → **security-review**
+- Capping request *rate/volume* per caller → **rate-limiting**
+- Recording *who did what when* for compliance/forensics → **build-audit-logging**
+- GDPR data-subject rights, lawful basis, PII mapping → **map-privacy-data-gdpr**
+- Fixing injection/XSS/SSRF in web code → **remediate-web-vulnerabilities**
+
+## Steps
+
+1. **Pick the model by the shape of the access rule — do not default to RBAC for everything.**
+
+   | Model | Decide by | Use when | Engine fit |
+   |---|---|---|---|
+   | **RBAC** | role → permission set | Fixed, coarse tiers (admin/editor/viewer); permissions don't depend on the specific row | DB tables, Casbin, Cedar |
+   | **ABAC** | attributes of subject+resource+context | Rules vary by field/status/time/region (`owner.dept == doc.dept AND time < embargo`) | **OPA/Rego**, Cedar |
+   | **ReBAC** | relationship/ownership graph | Per-resource sharing, nesting (`folder→doc`), "users this owner invited" — Drive/GitHub-style | **OpenFGA / SpiceDB** (Zanzibar), Oso |
+
+   Default: start **RBAC for app-wide roles**, add **ReBAC** the moment you need per-resource sharing or hierarchy, add **ABAC** conditions for field/context rules. They compose — RBAC roles can be relations in a ReBAC graph. Don't roll a bespoke nested-`if` engine; pick one of the named tools.
+
+2. **Separate authN from authZ — the decision is its own layer.** AuthN hands you a verified principal (`{user_id, tenant_id, roles}` from the validated token — see **auth-jwt-session**). AuthZ takes `(principal, action, resource)` → `allow|deny`. Never re-derive identity inside the policy, and never let the policy trust unverified claims.
+
+3. **Centralize the decision behind one `authorize()` call — never inline `if role ==`.** Every protected operation calls the same checkpoint; scattered checks drift and leak.
+
+   ```python
+   # ONE entry point. Engine (OPA/Cedar/Oso/OpenFGA) behind it.
+   def authorize(principal, action, resource):
+       decision = engine.check(
+           subject=principal.user_id,
+           tenant=principal.tenant_id,      # from token, NEVER from the request body
+           action=action,                   # "document:delete"
+           resource=resource,               # {id, type, tenant_id, owner_id, status}
+       )
+       if not decision.allow:               # deny by default — no rule matched = deny
+           raise Forbidden(action, resource.id)
+       return decision
+   ```
+
+4. **Enforce multi-tenant isolation on every query — and derive `tenant_id` from the token, never the client.** A client-supplied tenant/org id is an attacker-controlled cross-tenant key. Scope every read/write by the token's tenant; treat a missing tenant scope as a bug, not a default-all.
+
+   ```sql
+   -- Defense in depth: Postgres Row-Level Security so a forgotten WHERE can't leak.
+   ALTER TABLE documents ENABLE ROW LEVEL SECURITY;
+   ALTER TABLE documents FORCE ROW LEVEL SECURITY;          -- applies to table owner too
+   CREATE POLICY tenant_isolation ON documents
+     USING (tenant_id = current_setting('app.tenant_id')::uuid);
+   ```
+   Set `app.tenant_id` per request/connection from the verified token (`SET LOCAL app.tenant_id = ...` inside the request transaction). App-layer `WHERE tenant_id = $1` is the primary guard; RLS is the backstop for the day someone forgets it.
+
+5. **Deny by default, least privilege, deny wins.** No matching allow rule ⇒ deny. Start every role at zero permissions and add. When allow and deny rules overlap, **explicit deny beats allow**. Write this into the policy, don't rely on convention.
+
+6. **Make policy versioned, code-reviewed, and unit-tested — policy-as-code.** Keep `.rego` / Cedar / `policy.polar` in the repo, PR-reviewed like app code. Example OPA/Rego with the three non-negotiables baked in:
+
+   ```rego
+   package authz
+   default allow := false                          # deny by default
+   default deny := false                           # bare `deny` is always defined
+
+   allow if {                                       # owner can do anything to own resource
+     input.resource.tenant_id == input.principal.tenant_id   # same-tenant gate, always
+     input.resource.owner_id == input.principal.user_id
+   }
+   allow if {                                       # role grants the action
+     input.resource.tenant_id == input.principal.tenant_id
+     some role in input.principal.roles
+     grants[role][_] == input.action               # e.g. grants.editor[] = "document:edit"
+   }
+   deny if input.resource.status == "locked"        # explicit deny condition
+   final_allow := allow and not deny                # deny wins over any allow
+   ```
+   Wire the API to read `final_allow`, not `allow`. Run `opa test policy/ -v` in CI. The same input schema feeds both the running engine and the tests.
+
+7. **Pass the decision an explicit resource snapshot, fetched tenant-scoped first.** Load the resource (already filtered by tenant in the query) before checking, so the policy sees real `owner_id`/`status`/`tenant_id`. Checking by id alone, then fetching unscoped, reintroduces the IDOR.
+
+8. **Verify with an allow/deny matrix per role × action — including explicit cross-tenant denial** (see Verify) before shipping.
+
+## Common Errors
+
+- **Trusting a client-supplied `tenant_id`/`org_id`** from body, query, or header. It's the cross-tenant skeleton key. Derive tenant solely from the verified token; ignore any tenant field in the request.
+- **IDOR — checking the role but not the ownership/tenant of *this* row.** `can_edit_documents` is true, but the doc belongs to another tenant. Always bind the check to the specific resource's `tenant_id`/`owner_id`, fetched tenant-scoped.
+- **Inline `if user.role == 'admin'` scattered across handlers.** They drift, one gets missed, and a new action ships unguarded. Route every check through the single `authorize()` checkpoint.
+- **Role explosion (`editor_us_finance_readonly`).** Combinatorial roles that should be attributes. Move per-field/context rules to ABAC conditions; keep roles coarse.
+- **Allow-by-default / "fail open."** A request that matches no rule slips through, or an engine error returns allow. Set `default allow := false` and treat engine errors/timeouts as deny.
+- **Reading `allow` instead of the deny-wins result.** Exposing `allow` to the API skips the explicit-deny rule. Have the engine return `final_allow` (`allow and not deny`) so a locked/blocked resource can't be reached through a permissive role.
+- **AuthZ in the frontend only.** Hiding a button is UX, not security — the API is the enforcement boundary. Every server endpoint authorizes independently.
+- **Roles baked into the JWT and never refreshed.** Revoking a role doesn't take effect until the token expires. Check permissions against current state (or keep token TTL short and re-resolve roles server-side).
+- **No DB-tier backstop.** One forgotten `WHERE tenant_id` leaks every tenant. Enable Postgres RLS with `FORCE` so the data tier denies even when the app forgets.
+- **Confused-deputy / unscoped service calls.** A worker or internal service queries with god privileges on behalf of a user without carrying the user's tenant/permission scope. Propagate the principal; don't let internal callers bypass `authorize()`.
+- **Policy with no tests.** Untested Rego/Cedar rots silently. Ship the allow/deny matrix as `opa test` cases alongside the policy.
+
+## Verify
+
+1. **Allow/deny matrix — every role × action.** For each role (admin/editor/viewer/none) × each action (create/read/update/delete/share), assert the decision matches the intended table. Every cell is a test case, run in CI (`opa test policy/ -v` or the engine's harness).
+2. **Cross-tenant denial (the critical one).** User in tenant A requests a resource in tenant B by its real id → **403/deny**, for *every* action, including read. Do this both through the API and by querying the DB with `app.tenant_id` set to A — RLS must return zero rows.
+3. **IDOR probe.** As a non-owner same-tenant user, attempt update/delete on a resource you don't own and your role doesn't permit → deny. Then as owner → allow. Confirms the check binds to the resource, not just the role.
+4. **Deny by default.** Invent a brand-new action string with no policy rule → deny (not allow). Proves nothing slips through unmatched.
+5. **Deny wins.** A resource in `status = "locked"` (or a user under an explicit deny) → deny even when a role would otherwise allow. Assert against `final_allow`, the value the API consumes.
+6. **RLS backstop.** Run a `SELECT` that *omits* the app-layer tenant filter against a session with `app.tenant_id` set → still returns only that tenant's rows. Proves the data tier holds when the app forgets.
+7. **Centralization.** `grep -rnE 'role *== *|isAdmin|\.role\b' src/` finds zero authorization branches outside the policy layer — every decision goes through `authorize()`.
+8. **Privilege escalation negative test.** A user cannot grant themselves a role/permission or modify a policy they shouldn't (the "edit roles" action is itself authorized).
+
+Done = the role × action matrix passes in CI, cross-tenant and IDOR probes are denied at both the API and DB tier (RLS enforced), the policy is versioned with `default allow := false` and deny-wins (API reads `final_allow`), and `grep` finds no authorization logic outside the centralized layer.

package/skills/design-backup-dr-recovery/SKILL.md

@@ -0,0 +1,113 @@
+---
+name: design-backup-dr-recovery
+description: Designs and validates backup, point-in-time-recovery, and disaster-recovery strategy for datastores — sets RPO/RTO targets, configures snapshot plus continuous WAL/binlog/oplog archiving for PITR, 3-2-1 immutable retention, automated test-restores, and cross-region replica failover with split-brain fencing.
+when_to_use: When a stateful service needs a credible answer to "what if the database is lost or corrupted" — setting RPO/RTO, wiring snapshots + continuous log archiving for PITR, designing cross-region failover, scheduling tested restores, or auditing a never-restore-tested backup. Distinct from db-migration-safety (forward schema change safety) and incident-response-sre (running the live outage, not designing recoverability).
+---
+
+## When to Use
+
+Reach for this skill when the question is **"can we get the data back, and how fast"** — not how to change the schema:
+
+- "Set RPO/RTO for this database and prove we can hit them"
+- "We have nightly snapshots but no way to restore to 2:47pm — add PITR"
+- "Stand up cross-region DR / a warm standby we can promote"
+- "Our backups have never been restore-tested — audit and fix that"
+- "Recover a single dropped table without rolling back the whole DB"
+- "Defend backups against ransomware / a fat-fingered `DROP DATABASE`"
+
+NOT this skill:
+- Making a forward schema migration safe/reversible (expand-contract, online DDL) → db-migration-safety
+- Running the live incident — paging, comms, mitigation timeline → incident-response-sre
+- Protecting/rotating the backup-store credentials and KMS keys → secrets-management
+- Alerting that a backup job failed / dashboards for restore lag → observability-instrument
+- Trimming snapshot/storage spend → cloud-cost-optimize
+
+## Steps
+
+1. **Set RPO and RTO per datastore from business impact — these two numbers drive every later choice.** RPO = max tolerable data loss (how far back you may rewind). RTO = max tolerable downtime (how long restore may take). Pick a tier, don't invent per-DB:
+
+   | Tier | Example data | RPO | RTO | Implied mechanism |
+   |---|---|---|---|---|
+   | Tier 0 (money/orders) | payments ledger, auth | ≤ seconds | ≤ minutes | sync replica + continuous WAL, automated promotion |
+   | Tier 1 (core app) | primary OLTP DB | ≤ 5 min | ≤ 1 hr | snapshot + async WAL archiving (PITR), warm standby |
+   | Tier 2 (supporting) | analytics, search index | ≤ 1 hr | ≤ 4 hr | hourly snapshot, rebuild-from-source allowed |
+   | Tier 3 (derived/cache) | caches, rebuildable views | n/a | n/a | no backup — document the rebuild procedure instead |
+
+   RPO ≤ snapshot interval is a lie unless you also archive logs continuously (step 2). Write the chosen numbers down; an untargeted "we back up nightly" has an implicit 24h RPO nobody agreed to.
+
+2. **Two backup layers: periodic base + continuous log archiving. Snapshot-only cannot do PITR.** A snapshot gets you to *snapshot time*; the log stream replays forward to any timestamp in between.
+
+   | Engine | Base backup | Continuous log (the PITR engine) | Restore = base + replay |
+   |---|---|---|---|
+   | PostgreSQL | `pg_basebackup` / disk snapshot | WAL via `archive_command` → object store (pgBackRest/WAL-G) | `restore_command` + `recovery_target_time` |
+   | MySQL/MariaDB | `xtrabackup` / `mysqldump` | binlog (`log_bin`, `binlog_format=ROW`) shipped off-host | restore base, `mysqlbinlog --stop-datetime` apply |
+   | MongoDB | `mongodump` / filesystem snapshot | oplog (replica set required) | restore + `--oplogReplay --oplogLimit` |
+   | SQLite | `.backup` / file copy | WAL file is local-only — ship full DB on a cron | copy file (no true PITR) |
+   | Managed (RDS/Cloud SQL) | automated snapshots | provider-managed transaction logs | "restore to point in time" API |
+
+   Default for any Tier 0/1 SQL store: **pgBackRest/WAL-G (Postgres) or Percona XtraBackup + binlog (MySQL)** with logs archived every ≤60s. Logical dumps (`pg_dump`/`mysqldump`) are a *secondary* portable copy, not your primary — they're slow to restore and lock/strain a large live DB.
+
+3. **Retention and layout: 3-2-1 with at least one immutable copy.** 3 copies, 2 media/accounts, 1 off-site/cross-region. Make ≥1 copy **immutable** so ransomware or a compromised admin can't delete it:
+   - Object-lock the bucket: S3 Object Lock **Compliance mode** (`--object-lock-mode COMPLIANCE`), or GCS bucket retention lock, or Azure immutable blob. Compliance mode = nobody, including root, can delete before expiry.
+   - Put backups in a **separate account/project** from production with write-only (no-delete) IAM for the backup writer — same-account backups die with the account.
+   - Lifecycle: hot (last 7d, fast restore) → warm (30d) → cold/Glacier (90–365d per compliance). Cold tiers add hours to RTO — never put your RTO-critical recent backups in Glacier.
+   - Retention must cover **detection lag**: corruption found on day 10 needs a day-9 good copy, so retain > realistic time-to-detect.
+
+4. **Verify restorability automatically — an untested backup is a hypothesis, not a backup.** Schedule a job that restores to a *scratch* environment and validates, on every backup or at least nightly:
+   ```bash
+   # nightly restore drill (Postgres / pgBackRest), exits non-zero on any failure
+   pgbackrest --stanza=main --type=time \
+     --target="$(date -u -d '10 min ago' +'%Y-%m-%d %H:%M:%S')" restore
+   pg_ctl start -D "$PGDATA" -w -t 600
+   # validate: structural + content, not just "it started"
+   psql -tAc "SELECT count(*) FROM orders"            | grep -qE '^[0-9]+$'
+   psql -tAc "SELECT pg_catalog.pg_database_size('app')"   # > 0
+   RESTORE_SECS=$SECONDS; echo "restore took ${RESTORE_SECS}s (RTO budget: 3600s)"
+   [ "$RESTORE_SECS" -le 3600 ] || { echo "RTO BREACH"; exit 1; }
+   ```
+   Validate **content** (row counts vs a known watermark, `pg_amcheck`/`CHECKSUM TABLE`, app-level invariant query), measure wall-clock restore time, and **fail the job loud** (page) if it breaks or exceeds RTO. The restore time you measure here *is* your real RTO — the planned number is fiction until measured.
+
+5. **Have a procedure for each recovery shape — they are not the same command.**
+   - **Full restore (host lost):** provision, restore latest base, replay logs to "now", re-point app.
+   - **PITR (bad deploy/poison write at 14:32):** restore base before 14:32, replay to `recovery_target_time = '14:31:59'`, `pause_at_recovery_target=on`, inspect, then promote. Recover to *just before* the bad event.
+   - **Single-table / logical restore:** restore into a throwaway instance, `pg_dump -t orders` (or `mysqldump --no-create-info`) that table, load into prod — never restore the whole cluster to fix one table.
+   - **Corruption:** do **not** overwrite the only good copy. Restore to a new instance, run `pg_amcheck`/`mongod --repair`/`CHECK TABLE`, diff, cut over only after validation. Promote a healthy replica only after confirming the corruption didn't already replicate.
+
+6. **Cross-region/replica DR: pick sync vs async deliberately, and fence against split-brain.**
+
+   | | Sync replication | Async replication |
+   |---|---|---|
+   | RPO | ~0 (no committed loss) | replica lag (seconds–minutes) |
+   | Write latency | + cross-region RTT every commit | none (local commit) |
+   | Use for | Tier 0 only, regions < ~10ms apart | everything else (default) |
+
+   Default to **async** unless RPO≈0 is mandated and you accept the write-latency tax. Failover = promote standby + cut traffic over. **Split-brain is the real danger**: if the old primary comes back and also takes writes, you get divergent histories that can't be merged. Enforce a quorum/leader-election (Patroni + etcd/Consul, Orchestrator, or RDS Multi-AZ which fences for you) and **STONITH-fence** the old primary (revoke its network/credentials) *before* promoting. Cut traffic via low-TTL DNS (≤30s) or, better, a connection proxy (PgBouncer/HAProxy/ProxySQL) that flips backends instantly — DNS TTL caching makes raw DNS failover slow and uneven.
+
+7. **Write the runbook with exact commands, and rehearse it (game day).** The runbook lists per scenario: detection signal → exact restore/promote commands (copy-pasteable, with placeholders) → validation queries → traffic-cutover step → rollback-of-the-rollback. Store it **outside** the system it recovers (it's useless if it lives only in the DB that's down). Schedule a **DR drill quarterly** (Tier 0: monthly) that actually fails over to the standby/restored copy under timing — measure RTO/RPO against target, file the gaps. A runbook never executed end-to-end is presumed broken.
+
+## Common Errors
+
+- **Never restore-testing.** The #1 cause of "we had backups but couldn't recover." A backup that has never been restored is unproven; automate the drill (step 4) so success/failure is observed continuously, not discovered during the outage.
+- **Snapshot-only, calling it PITR.** Nightly snapshots = up to 24h RPO and you can only land on snapshot boundaries. PITR requires continuous WAL/binlog/oplog archiving (step 2). If asked for "restore to any second," snapshots alone cannot.
+- **Same blast radius.** Backups in the same account/region/bucket as prod die with it — one compromised credential, one region outage, one `DROP` and both the data and its backup are gone. Cross-account + cross-region + immutable is the point.
+- **No immutability → ransomware/insider wipes the backups too.** Mutable backups are deleted in the same attack that hit prod. Use object-lock Compliance mode / retention lock on ≥1 copy.
+- **Replica treated as a backup.** A replica faithfully replicates `DELETE FROM users` and corruption in milliseconds. Replication is for availability/failover; it is **not** a backup and gives zero protection against logical errors. You need both.
+- **Logical dump as the primary backup for a large DB.** `pg_dump`/`mysqldump` of a multi-TB DB takes hours to restore and strains/locks the live DB while running — blows RTO. Use physical base + log archiving; keep logical dumps as a secondary portable copy only.
+- **RTO ignores restore *and* warm-up.** Real RTO = provision + transfer + restore + log replay + cache/index warm-up + cutover. Cold-tier (Glacier) retrieval alone can be hours. Measure end-to-end; don't quote the `restore` command's runtime.
+- **Failover with no split-brain fencing.** Promoting a standby while the old primary still accepts writes forks history irrecoverably. Fence (STONITH) the old primary and use quorum-based promotion before flipping traffic.
+- **DNS-only cutover with long TTL.** A 300s+ TTL means clients keep hitting the dead primary long past promotion. Use TTL ≤30s, or a connection proxy that switches backends instantly.
+- **Backup job "succeeds" but the file is empty/corrupt.** Exit-0 ≠ valid backup. Verify object size > expected floor, checksum, and a test-restore — not just the job's return code.
+- **Retention shorter than detection lag.** Corruption noticed on day 10 with 7-day retention = no clean copy exists. Retain past your realistic time-to-detect, and keep a longer-interval cold copy.
+
+## Verify
+
+1. **RPO/RTO are written and tiered.** Every stateful datastore has an explicit RPO and RTO number tied to a business tier (step 1) — not an implicit "nightly."
+2. **PITR proven, not assumed.** Restore to an *arbitrary* timestamp between two base backups (e.g. 14:31:59 yesterday) lands the data at that second — proves continuous log archiving works, not just snapshots.
+3. **Automated restore drill is green and timed.** The nightly/per-backup test-restore to scratch passes (structural + content + invariant checks) and its measured wall-clock ≤ RTO budget; a failure or RTO breach **pages**.
+4. **3-2-1 + immutability holds.** ≥3 copies across ≥2 accounts/regions, ≥1 with object-lock Compliance/retention-lock that even root cannot delete before expiry — confirm by attempting (and failing) to delete a locked object.
+5. **Independent blast radius.** Deleting/encrypting the prod bucket/account leaves a usable backup intact in another account/region.
+6. **Each recovery shape has a tested path:** full restore, PITR-to-timestamp, single-table logical restore, and corruption-to-new-instance — each with copy-pasteable commands in the runbook.
+7. **Failover fences and cuts over fast.** A drill promotion fences the old primary (it cannot take writes post-promotion) and traffic moves via ≤30s-TTL DNS or a proxy; no split-brain divergence after.
+8. **Game day actually ran.** A dated DR drill within the cadence (≤1 quarter; Tier 0 ≤1 month) failed over end-to-end, measured RPO/RTO vs target, and logged the gaps.
+
+Done = every datastore has written RPO/RTO targets, PITR (base + continuous logs) restoring to an arbitrary timestamp, an automated restore drill that is green and within RTO, ≥1 immutable cross-account/region copy, and a runbook proven by a dated end-to-end DR drill — restore time **measured**, never merely planned.