sanook-cli 0.4.0 → 0.5.1

package/skills/send-transactional-email/SKILL.md

@@ -0,0 +1,126 @@
+---
+name: send-transactional-email
+description: Ships reliable transactional email (password resets, receipts, verification, alerts) where the hard part is deliverability, not the API call — authenticate the From domain with SPF/DKIM/DMARC alignment, send through a provider (SES/Postmark/SendGrid/Resend/Mailgun) instead of a cold self-hosted MTA, isolate transactional from marketing streams, build inlined-CSS multipart emails, send idempotently via a job runner, and process bounce/complaint webhooks into a suppression list so mail actually lands in the inbox.
+when_to_use: Sending or fixing delivery of transactional email — auth/verification/reset/receipt mail landing in spam, domain authentication (SPF/DKIM/DMARC), bounce/complaint handling, suppression lists, or rendering. Distinct from implement-push-notifications (the mobile/web PUSH channel, a different transport entirely) and message-queue-jobs (the async job system that ENQUEUES the send and owns retry/DLQ — this skill owns the email-specific deliverability, content, and feedback loop).
+---
+
+## When to Use
+
+Reach for this skill when the work is **getting a transactional email into the inbox and reacting to what bounces** — domain auth, provider routing, content, and the feedback loop:
+
+- "Password-reset / verification emails are landing in spam (or vanishing) — fix deliverability"
+- "Set up SPF / DKIM / DMARC so our From domain authenticates and aligns"
+- "Pick and wire a provider (SES, Postmark, SendGrid, Resend, Mailgun) for receipts/alerts"
+- "Our marketing blasts are tanking password-reset delivery — separate the streams"
+- "Process bounce + complaint webhooks and stop re-sending to dead addresses"
+- "Build the email so it renders right in Outlook/Gmail/dark mode with a plain-text fallback"
+- "A retry double-sent the receipt / verification email — make sends idempotent"
+
+NOT this skill:
+- The async job/queue that **enqueues** the send, owns retry-with-backoff, DLQ, poison-message handling → message-queue-jobs (this skill is what runs *inside* that job)
+- The idempotency-key store/dedup primitive that makes the enqueue+send exactly-once → idempotency-keys
+- Mobile/web **PUSH** notifications (APNs/FCM/Web Push) — a different transport, not email → implement-push-notifications
+- The raw DNS record mechanics (TTL, zone editing, how a TXT/CNAME is published) → configure-dns-tls (this skill tells you *which* records; that skill publishes them)
+- Tracking-pixel/open-tracking consent, unsubscribe-data handling, PII retention/erasure → map-privacy-data-gdpr
+- Throttling how many emails one user can trigger → rate-limiting
+- Marketing campaigns, newsletters, drip sequences, segmentation → (out of scope — a different sending stream entirely; see step 3)
+
+This skill owns **domain authentication, provider/stream choice, email content, idempotent sending, and feedback processing**. It hands the actual job-running to message-queue-jobs.
+
+## Steps
+
+1. **Authenticate the sending domain — this is the gate, not optional.** Gmail/Yahoo require SPF + DKIM + DMARC on bulk and increasingly on all mail; without alignment you go to spam or get rejected. Publish all three on the From domain (records owned by configure-dns-tls; *values* below). Use a dedicated subdomain like `mail.example.com` / `txn.example.com` so reputation is scoped.
+
+   | Record | Where | Value (shape) | Purpose |
+   |---|---|---|---|
+   | **SPF** | `TXT` at sending domain | `v=spf1 include:amazonses.com ~all` (one TXT, ≤10 DNS lookups, `~all` not `-all` until verified) | authorizes the provider's IPs in `Return-Path` |
+   | **DKIM** | provider-given `CNAME`s (SES, Resend) or `TXT` (`<sel>._domainkey`) | provider publishes the public key; mail is signed `d=example.com` | cryptographic signature, survives forwarding |
+   | **DMARC** | `TXT` at `_dmarc.example.com` | `v=DMARC1; p=none; rua=mailto:dmarc@example.com; adkim=s; aspf=s` | tells receivers what to do on auth fail + reports |
+
+   **Alignment is the part people miss:** DMARC passes only if SPF *or* DKIM passes **and** its domain matches the **visible `From:`** domain. `Return-Path: bounces@provider.com` aligning SPF to the provider does **not** align to your From — so DKIM `d=` must equal your From domain. Set a **custom Return-Path / MAIL FROM** subdomain (`bounce.example.com`) at the provider for SPF alignment too. Roll DMARC `p=none` → monitor `rua` reports for 1–4 weeks → `p=quarantine` → `p=reject`. Never start at `reject`; you'll blackhole your own mail.
+
+2. **Send through a reputable provider — do NOT run your own SMTP MTA on cold IPs.** A fresh cloud IP has zero reputation and is often already on a blocklist; running Postfix yourself means you own PTR, warmup, FBL enrollment, and blocklist fights. Use a provider:
+
+   | Provider | Best for | Notes |
+   |---|---|---|
+   | **Postmark** | pure transactional, fastest inbox | hard-blocks marketing on transactional streams; great deliverability |
+   | **Amazon SES** | volume, cost | cheapest; you do more setup; sandbox until prod access granted |
+   | **Resend** | DX-first, modern stacks | React-email native; simple DKIM CNAMEs |
+   | **SendGrid / Mailgun** | scale, both streams | bigger surface, more knobs |
+
+   If you self-host anyway (rare): set **PTR / reverse DNS** so the IP resolves back to your HELO hostname (no PTR ≈ instant spam), enroll in every provider's **FBL**, and warm the IP. For 99% of cases, a provider is the answer.
+
+3. **Separate TRANSACTIONAL from MARKETING — different subdomains, IPs, and streams.** A marketing complaint must **never** be able to poison password-reset delivery. Use `txn.example.com` (or a dedicated transactional stream/IP pool) for resets/receipts/verification, and `news.example.com` (separate IP/stream) for campaigns. Postmark enforces this with separate Streams; SES uses separate **configuration sets** + dedicated IP pools. Mixing them means one bad newsletter tanks your ability to log users in.
+
+4. **Dedicated vs shared IP, and warm up before volume.** Shared IP (provider's pool) is fine and *better* at low/spiky volume — you inherit the pool's warm reputation. Move to a **dedicated IP** only above ~100k/month steady, then **warm it**: ramp send volume gradually so receivers learn the IP is legit.
+
+   | Day | Max sends/day (rough) |
+   |---|---|
+   | 1–2 | 50 → 100 |
+   | 3–5 | 500 → 1,000 |
+   | 6–10 | 5,000 → 20,000 |
+   | 11–20 | double daily toward target |
+
+   Send your **best, most-engaged traffic first** during warmup; complaints early on a cold dedicated IP are very expensive.
+
+5. **Build the email so it actually renders — inline CSS, multipart, dark-mode, accessible.** Email clients (esp. Outlook/Word engine, Gmail) strip `<style>`, ignore flexbox/grid, and need table layout. Use **MJML** (compiles to bulletproof tables) or a templating tool with a **CSS inliner** (`juice`, premailer) — never raw `<div>` flexbox.
+   - **Always send `multipart/alternative`** with both `text/plain` AND `text/html`. A missing/empty plain-text part is a strong spam signal and breaks watches/screen readers.
+   - **Inline every style** (`style="…"` on elements); media queries in `<head>` for mobile/dark-mode are progressive enhancement only.
+   - **Dark mode:** set `<meta name="color-scheme" content="light dark">` and `supported-color-schemes`; don't rely on transparent PNG logos (add a background).
+   - **Accessible:** real `alt` on images (many clients block images by default — the email must make sense with images off), sufficient contrast, semantic headings, descriptive link text (not "click here").
+   - Put the critical action (reset link, code) in **text**, not baked into an image.
+
+6. **Set From / Reply-To / Return-Path correctly.** `From:` = a real, branded, *authenticated* address on your sending domain (`noreply@txn.example.com` is fine but a monitored `Reply-To` is friendlier). `Reply-To:` → where humans actually reach you (`support@example.com`). **`Return-Path` / envelope MAIL FROM** → the provider's/your bounce-handling address on an SPF-aligned subdomain; this is where bounces go and what SPF checks — **never** your visible From. Mismatched/spoofed From domains fail DMARC.
+
+7. **Make every send idempotent — a retry must not double-send.** The job runner (message-queue-jobs) will retry on transient failure; without a guard, the user gets two receipts. Compute a stable **idempotency key** per logical email (e.g. `sha256(user_id + email_type + event_id)`) and record it transactionally before/with the send. Most providers also accept a request-level idempotency/dedup token — pass it. (The dedup-store primitive is idempotency-keys; this skill defines *what makes an email send unique*.)
+
+   ```python
+   key = sha256(f"{user_id}:password_reset:{reset_request_id}").hexdigest()
+   if not claim_idempotency_key(key):      # atomic INSERT … ON CONFLICT DO NOTHING
+       return                              # already sent — silently no-op
+   provider.send(msg, idempotency_key=key) # provider-level dedup too
+   ```
+   Enqueue the send as a job rather than sending inline in the request path, so a slow provider or 5xx doesn't fail the user's HTTP request — see message-queue-jobs.
+
+8. **Process bounces + complaints and maintain a suppression list — never re-send to dead/complained addresses.** Wire the provider's **webhooks** (SES→SNS, Postmark/SendGrid/Mailgun event webhooks) and feed a `suppression` table that the send path checks *before* every send. Verify webhook signatures (these are untrusted inbound — see ingest-webhook-secure).
+
+   | Event | Meaning | Action |
+   |---|---|---|
+   | **Hard bounce** | address doesn't exist | **suppress permanently**, never retry |
+   | **Soft bounce** | mailbox full / temporary | retry a few times, then suppress if persistent |
+   | **Complaint (FBL)** | user hit "spam" | **suppress permanently**; investigate — this is reputation poison |
+   | **Spam / blocked** | content/IP blocked | pause stream, inspect content/reputation |
+
+   A single complaint costs far more than a lost email. Re-sending to a hard bounce or complainer destroys sender reputation for *everyone* on the stream.
+
+9. **Honor unsubscribe — `List-Unsubscribe` + One-Click — even on transactional.** Gmail/Yahoo bulk rules require a `List-Unsubscribe` header with **one-click** support; even for transactional mail it's good practice (and required if there's any promotional content). Pure system mail (password reset) can be exempt, but adding the header never hurts.
+
+   ```
+   List-Unsubscribe: <https://example.com/u/abc123>, <mailto:unsub@example.com>
+   List-Unsubscribe-Post: List-Unsubscribe=One-Click
+   ```
+   A POST to the URL must unsubscribe with no further interaction. (Consent/unsubscribe *data* handling → map-privacy-data-gdpr.)
+
+10. **Monitor reputation and stay under the thresholds.** Enroll the domain in **Google Postmaster Tools** and watch your provider's dashboards. Hard limits that get you throttled/blocked: **complaint rate < 0.1%** (Gmail's red line is 0.3%, but treat 0.1% as the ceiling), bounce rate low single digits, no blocklist hits. Set alerting on a spike (observability-instrument). A climbing complaint rate is an early warning before a hard block.
+
+11. **Test in a sandbox — NEVER send to real addresses from staging.** Catch a misconfigured loop emailing 50k real users *before* prod.
+    - **Local/CI:** capture all SMTP into **Mailpit** or **MailHog** (a fake inbox); assert subject, both MIME parts, and rendered HTML in tests.
+    - **Provider sandbox:** **SES sandbox** only delivers to verified addresses; Postmark has a test API token that accepts-but-doesn't-deliver.
+    - **Inbox placement / seed list:** before a big change, send to a seed list (GlockApps/provider tools) to see Gmail/Outlook/Yahoo inbox-vs-spam placement.
+    - Gate the real provider behind an env flag so staging can only hit Mailpit/sandbox — never live SMTP.
+
+12. **Mind tracking privacy and don't trust open rates.** Open tracking = a 1×1 pixel; **Apple Mail Privacy Protection (MPP)** pre-fetches it, **inflating opens to near-100%** and making opens worthless for engagement. Tracking pixels are personal-data processing under GDPR — needs a lawful basis and arguably consent (→ map-privacy-data-gdpr). For transactional mail, prefer **no open tracking**; if you wrap links for click tracking, keep redirects fast and on your own domain so they don't trip spam filters or break the link on failure.
+
+## Verify
+
+1. **Auth passes and aligns:** send to `check-auth@verifier.port25.com` or mail-tester.com — SPF `pass`, DKIM `pass` with `d=` your From domain, DMARC `pass` with **alignment**. `dig TXT _dmarc.example.com` shows the policy; `dig TXT <sel>._domainkey.example.com` resolves.
+2. **DMARC ramped safely:** `rua` aggregate reports show your legit mail passing for 1–4 weeks at `p=none` *before* you move to `quarantine`/`reject`.
+3. **Streams isolated:** a forced complaint/bounce on the marketing stream does **not** appear in or degrade the transactional stream's reputation/dashboards.
+4. **Renders everywhere:** the HTML shows correctly in Gmail, Outlook (Word engine), Apple Mail, and dark mode; with images blocked the email is still actionable (alt text, text link); a `text/plain` part exists and is non-empty.
+5. **Idempotent:** trigger the same logical email twice (or force a job retry) → exactly **one** message is delivered; the second is a no-op.
+6. **Feedback loop works:** send to a provider seed/simulator bounce + complaint address → webhook fires, the address lands in the **suppression** table, and a subsequent send to it is **skipped before** hitting the provider.
+7. **Unsubscribe one-click:** a POST to the `List-Unsubscribe` URL unsubscribes with no extra step; Gmail shows the unsubscribe affordance.
+8. **No real mail from non-prod:** staging/CI sends are captured by Mailpit/MailHog/sandbox and cannot reach a real inbox; a deliberate "send to a real address" from staging is blocked.
+9. **Reputation green:** Google Postmaster shows domain reputation High/Medium, complaint rate **< 0.1%**, no blocklist entries.
+
+Done = the From domain passes SPF/DKIM/DMARC with alignment (DMARC ramped p=none→quarantine→reject on real report data), transactional mail goes through a provider on a stream isolated from marketing, emails render with inlined CSS + a plain-text part, sends are idempotent under retry, bounces/complaints flow into a suppression list that the send path honors, and no staging environment can email a real user.

package/skills/serve-deploy-ml-model/SKILL.md

@@ -0,0 +1,107 @@
+---
+name: serve-deploy-ml-model
+description: Deploys a trained ML model to production — packaging it with the identical training-time preprocessing, registering a versioned model+code+data triple, serving via batch or online REST/gRPC behind a runtime (BentoML/TorchServe/Triton/ONNX), with autoscaling/warmup and canary/shadow rollout — so served predictions reproducibly match offline scoring.
+when_to_use: Taking a trained model to production to generate predictions (package, register, serve, scale, roll out). Distinct from train-evaluate-ml-model (building/evaluating the model), monitor-ml-drift (post-deployment drift/quality monitoring), and deploy-release (generic application deploys with no model artifact).
+---
+
+## When to Use
+
+Reach for this skill when a working model needs to **serve predictions in production**, not when it's still being built:
+
+- "Deploy this model so the app/service can call it"
+- "Stand up a REST/gRPC inference endpoint for `model.pkl`/`model.pt`"
+- "Run nightly batch scoring over the warehouse table"
+- "Roll the new model out behind the old one (shadow/canary) before cutting over"
+- "Our predictions in prod don't match what we got in the notebook" (train/serve skew)
+- "Speed up / scale the inference service" (ONNX export, autoscaling, warmup)
+
+NOT this skill:
+- Training, hyperparameter search, offline metrics, choosing the model → train-evaluate-ml-model
+- Watching the *live* model for input drift, label delay, quality decay, alerting → monitor-ml-drift
+- Shipping a normal app/service with no model artifact (web app, API, worker) → deploy-release
+- Percentage ramps, kill switches, sticky bucketing for *any* change → feature-flags-rollout (this skill uses it for the model rollout)
+- Latency/cost tuning of an *LLM* prompt/provider path → optimize-llm-cost-latency
+
+## Steps
+
+1. **Package the model WITH its exact preprocessing — this is the #1 cause of train/serve skew.** The artifact must contain the *same* feature/transform code that produced training inputs, not a reimplementation. Fit transforms on train data, serialize the fitted objects, and apply the identical pipeline at serve time.
+
+   ```python
+   # train.py — ONE fitted pipeline = preprocessing + model, saved as a unit
+   from sklearn.pipeline import Pipeline
+   from sklearn.compose import ColumnTransformer
+   import mlflow, mlflow.sklearn
+
+   pipe = Pipeline([("prep", ColumnTransformer(...)), ("model", clf)]).fit(X_tr, y_tr)
+
+   with mlflow.start_run():
+       mlflow.sklearn.log_model(
+           pipe, "model",
+           registered_model_name="churn",
+           input_example=X_tr.iloc[:5],          # captures schema + dtypes
+           signature=mlflow.models.infer_signature(X_tr, pipe.predict(X_tr)),
+           pip_requirements="requirements.lock",  # pinned, == training env
+       )
+   ```
+   Rules: never re-derive features in the serving codebase; serve the fitted `prep+model` as one object. For deep nets, save the transform graph (e.g. `torchvision`/`torchaudio` transforms or a `tf.function` preprocessing layer) *inside* the exported module so the runtime applies it. Stateful features (counts, embeddings, aggregates) computed from a feature store at train time must be read from the **same** store online — recomputing them in app code drifts.
+
+2. **Register and pin model + code + data together.** A model version is meaningless without the code and data snapshot that produced it. Push to a registry (MLflow Model Registry, SageMaker, Vertex, or a tagged OCI artifact) and record, in the run/version metadata: git SHA, training-data version/hash (DVC/Delta/snapshot id), and the locked dependency file. Use registry **stages** (`Staging` → `Production`) or aliases; deploy by *immutable version*, never "latest".
+
+3. **Pick the serving pattern by latency need — decide, don't hedge.**
+
+   | Pattern | Use when | Interface | Default runtime |
+   |---|---|---|---|
+   | **Batch / offline** | No realtime need; score a table/file on a schedule | Job writes predictions to warehouse/S3 | Spark / Ray / a plain container in cron/Airflow |
+   | **Online (sync)** | A user request blocks on the prediction; p99 budget < ~200 ms | **REST** (simple, debuggable) default; **gRPC** when p99 < 20 ms or high QPS | BentoML / TorchServe / Triton |
+   | **Streaming** | React to an event flow (clicks, transactions) continuously | Consume Kafka/Kinesis → predict → emit | Flink / Faust / a Ray Serve consumer |
+
+   Defaults: **batch unless something blocks on the result** — it's cheaper, simpler, and trivially reproducible. For online, start with **REST + JSON** and only move to gRPC/protobuf when a measured latency budget forces it. Do not build an online endpoint for a nightly report.
+
+4. **Choose the runtime; export to ONNX/TensorRT only when you need the speed.** Server defaults: **BentoML** (Python-first, easy custom logic, batching) for most teams; **Triton** for multi-framework, GPU, dynamic batching at scale; **TorchServe** for pure PyTorch shops. Convert to **ONNX Runtime** (CPU) or **TensorRT** (GPU) when profiling shows the framework runtime is the bottleneck — and **re-verify outputs match** the original within tolerance (atol≈1e-4) before trusting it; quantization/op-set changes silently alter predictions.
+
+   ```python
+   # bento service.py — load a PINNED model version (never "latest"), server-side batching
+   import bentoml
+   from bentoml.io import JSON
+   runner = bentoml.mlflow.get("churn:prod").to_runner()   # alias -> immutable version; never churn:latest
+   svc = bentoml.Service("churn", runners=[runner])
+
+   @svc.api(input=JSON(), output=JSON())   # set batchable=True + max_batch_size on the runner config for throughput
+   async def predict(rows: list[dict]) -> list[dict]:
+       return await runner.predict.async_run(rows)
+   ```
+
+5. **Add warmup, resource limits, and autoscaling — in that order.** Cold models cause p99 spikes: run a synthetic prediction at startup (load weights, JIT/CUDA-warm, fill caches) and gate the readiness probe on it so traffic only arrives warm. Set CPU/memory/GPU **requests and limits** from a load test (see load-stress-test), not by guessing. Autoscale on the right signal — **request concurrency / queue depth / GPU util**, not CPU% for GPU models — with `minReplicas ≥ 2` (no cold-start on scale-from-zero for latency-critical paths) and a scale-down stabilization window so it doesn't flap. Pin threads (`OMP_NUM_THREADS`) to avoid oversubscription under the container limit.
+
+6. **Roll out shadow → canary against the current model; keep an instant rollback.** Never hard-cut. **Shadow** first: mirror live traffic to the new version, log its predictions, serve the old model's response to users — compares behavior on real traffic at zero user risk. Then **canary**: route 1% → 10% → 50% → 100% by sticky hashed bucketing, watching guardrail metrics (latency, error rate, and prediction distribution vs the incumbent); auto-halt and revert on breach. Drive the ramp/kill switch with feature-flags-rollout. Rollback = repoint the alias/route to the previous **registered version** (still deployed) — must be one command, seconds, no rebuild.
+
+7. **Lock inference reproducibility end to end.** Serve from the **locked** requirements captured at registration (same library versions, same op-set), pin the base image by digest, set seeds where any stochasticity exists, and freeze the feature-store read path. The contract: the same input row produces a bit-identical (or within-tolerance) prediction in the notebook, the batch job, and the online endpoint.
+
+## Common Errors
+
+- **Reimplementing preprocessing in the serving code.** The serving normalizer/encoder/tokenizer drifts from the training one → skew. Serialize and serve the *fitted* pipeline as one artifact; never rewrite the transforms.
+- **Fitting a transform at serve time** (e.g. `StandardScaler().fit(request_batch)`, or imputing with the request's own mean). Must use stats fitted on **training** data, frozen in the artifact.
+- **Deploying "latest"/an unpinned stage.** A retrain silently swaps the model under prod. Deploy an immutable version id; promote via alias (`churn:prod`), not `churn:latest`.
+- **Env mismatch between train and serve.** Different numpy/sklearn/torch/CUDA or ONNX op-set changes outputs. Serve from the exact locked requirements; pin the image by digest.
+- **ONNX/TensorRT export assumed equivalent.** Quantization, fused ops, or op-set bumps shift predictions. Always diff converted vs original outputs on a fixed sample before shipping.
+- **No warmup → readiness flaps.** First requests hit an unloaded/un-JIT'd model and time out; the cold pod is added to the pool before it can serve. Warm at startup and gate readiness on it.
+- **Online endpoint for a batch problem.** Standing up a low-latency REST service to score a table on a schedule wastes cost and adds failure modes. Use a batch job.
+- **Hard cutover with no shadow/canary.** A skew or perf regression hits 100% of traffic instantly. Shadow, then ramp, with auto-rollback.
+- **Single replica / scale-to-zero on a latency path.** Any restart or scale event becomes a user-visible cold start. Keep `minReplicas ≥ 2`.
+- **Autoscaling GPU models on CPU%.** CPU sits low while the GPU saturates → it never scales and latency explodes. Scale on concurrency/queue depth/GPU util.
+- **Stateful features recomputed in app code.** Online aggregates/counts computed differently from the training feature store drift per request. Read from the same store.
+- **No rollback artifact.** The previous version was torn down, so "revert" means a rebuild. Keep the prior registered version deployed and one alias-flip away.
+
+## Verify
+
+1. **Parity (the skew gate):** Take a **fixed** holdout sample, score it three ways — training notebook, the batch job, and the online endpoint — and assert predictions match within tolerance (exact for classification labels; `atol≤1e-4` for probabilities/regression). Any mismatch blocks the deploy. This is the single most important check.
+2. **ONNX/quantized parity:** If exported, diff converted-runtime outputs vs the original framework on the same sample within tolerance.
+3. **Schema/contract:** Send a malformed/missing-field request → a clean 4xx, not a 500 or a silently wrong prediction. The logged input signature matches the registered one.
+4. **Latency/throughput:** Under the target arrival rate (load-stress-test), p95/p99 and sustained QPS meet the documented SLO **with warmup applied** — measure warm, not cold.
+5. **Warmup/readiness:** A freshly started replica reports ready only after a successful synthetic prediction; first real request is not a cold spike.
+6. **Autoscaling:** Drive load past the per-replica knee → replicas scale up on the chosen signal and back down after the stabilization window; `minReplicas` is honored at idle.
+7. **Shadow:** New version receives mirrored traffic and logs predictions while users still get the incumbent's response; their distributions are comparable before any canary.
+8. **Rollback:** Flip the alias to the previous version and confirm traffic serves the old model within seconds, no rebuild.
+9. **Reproducibility pin:** The deployed image digest, model version, training-data hash, and git SHA are all recorded together and resolvable from the running service.
+
+Done = served predictions match offline scoring on the fixed sample within tolerance, latency/throughput meet the SLO warm, shadow/canary ran with guardrails, and a one-command rollback to the prior registered version is proven.

package/skills/setup-cdn-edge-waf/SKILL.md

@@ -0,0 +1,107 @@
+---
+name: setup-cdn-edge-waf
+description: Configures CDN/edge delivery and a WAF — cache keys and Cache-Control/Surrogate-Control, stale-while-revalidate, tag/path purge wired into deploy, origin shielding with request collapsing, edge TLS + HTTP/3, and a managed OWASP ruleset with edge rate-limiting and bot/DDoS mitigation rolled out detect-then-enforce — to raise cache hit-ratio and absorb attacks before the origin.
+when_to_use: Serving static assets or APIs through a CDN, raising cache hit-ratio, or adding edge security (WAF, DDoS, bot mitigation). Distinct from caching-strategy (app/Redis caching), configure-reverse-proxy-lb (the origin proxy itself), rate-limiting (origin app limits), and configure-dns-tls (the cert/DNS records the CDN sits on top of).
+---
+
+## When to Use
+
+Reach for this skill when the work is at the **edge in front of the origin** — caching, TLS termination, and attack absorption at the POP, not inside your app:
+
+- "Put the static site / assets / images behind a CDN and stop hitting origin"
+- "Our cache hit-ratio is low / everything is `MISS` / origin is melting under read traffic"
+- "Cache this API response at the edge but purge it the instant we deploy / the record changes"
+- "Add a WAF / block SQLi+XSS / OWASP ruleset / managed rules"
+- "We're getting scraped / credential-stuffed / L7 DDoS'd — mitigate at the edge before it reaches us"
+- "Enable HTTP/3, edge TLS, origin shielding, request collapsing"
+
+NOT this skill:
+- Cache-aside / write-through / Redis TTLs *inside the application* → caching-strategy
+- Configuring the origin reverse proxy / load balancer (nginx/Envoy/HAProxy, upstream pools, health checks) → configure-reverse-proxy-lb
+- Per-user/per-key request quotas enforced *in the origin app* (429 + Retry-After business logic) → rate-limiting
+- Issuing the cert, DNS records, HSTS, CAA that the CDN hostname rides on → configure-dns-tls
+- Fixing the actual SQLi/XSS/SSRF in code (the WAF is a shield, not a patch) → remediate-web-vulnerabilities
+- Actively pen-testing the app to find injection bugs → fuzz-dynamic-security-test
+
+## Steps
+
+1. **Classify every route into a cache class first — caching policy follows the class, not the URL.** Decide this before touching any config.
+
+   | Class | Example | Cache-Control | Edge TTL | Cache key includes |
+   |---|---|---|---|---|
+   | Immutable static | `/_next/static/*.js`, hashed assets | `public, max-age=31536000, immutable` | 1y | path only |
+   | Versioned media | `/img/logo.png` | `public, max-age=86400, stale-while-revalidate=604800` | 1d | path only |
+   | HTML (anon) | `/`, `/blog/x` | `public, max-age=0, s-maxage=300, stale-while-revalidate=86400` | 5m (edge), 0 (browser) | path + `Vary` allow-list |
+   | Cacheable API (GET) | `/api/products` | `public, s-maxage=60, stale-while-revalidate=300` | 60s | path + canonical query + auth-tier |
+   | Private / auth | `/account`, `POST` | `private, no-store` | bypass | never cache |
+
+   Split `max-age` (browser) from `s-maxage`/`Surrogate-Control` (edge) so you can hold a long edge TTL while browsers revalidate. Origin sets `Surrogate-Control: max-age=...` (CDN strips it before responding); browser-facing `Cache-Control` carries the public value.
+
+2. **Engineer the cache key — this is where hit-ratio is won or lost.** Default to: `scheme + host + path + sorted-allowlisted-query`.
+   - **Strip tracking/garbage query params** (`utm_*`, `fbclid`, `gclid`, `ref`, session ids) from the key — otherwise every share link is a `MISS`. Allow-list the params that actually change the response; drop everything else.
+   - **Sort query params** so `?a=1&b=2` and `?b=2&a=1` collapse to one entry.
+   - **`Vary` only on what truly changes the body** — usually `Accept-Encoding` (or let the CDN normalize it) and at most a derived `X-Device-Type` (mobile/desktop) or auth-tier cookie you set. **Never `Vary: Cookie` or `Vary: User-Agent`** — that fragments the cache into near-uniqueness and kills hit-ratio.
+   - For cacheable APIs, normalize the auth dimension to a *tier* (e.g. `anon` vs `member`), not the raw token — fold the token down to a coarse bucket in an edge function before keying.
+
+3. **Add `stale-while-revalidate` and `stale-if-error` everywhere cacheable.** SWR serves the stale object instantly and revalidates in the background — no user waits on origin. `stale-if-error` keeps the site up when origin 5xx's. Without these, every TTL expiry is a latency spike and a thundering-herd to origin.
+
+4. **Wire purge into deploy — by tag/path, never blanket.** A full "purge everything" after each deploy nukes hit-ratio to ~0 and stampedes origin. Tag objects with surrogate keys at the origin and purge those keys:
+
+   ```http
+   # origin response tags the object
+   Surrogate-Key: product-42 catalog homepage
+
+   # deploy/webhook purges only the affected keys (Fastly example)
+   curl -X POST -H "Fastly-Key: $FASTLY_API_TOKEN" \
+        https://api.fastly.com/service/$SVC/purge/product-42
+   ```
+   - Static assets are **content-hashed** (`app.4f2a.js`) → never purge them, just deploy new filenames; old ones expire naturally.
+   - Purge HTML/API by surrogate key on the entity that changed (`product-42`), driven from the same event that writes the DB — not on a timer.
+   - Reserve path-based soft purge for the rare untagged route. Blanket purge is a break-glass action, not a deploy step.
+
+5. **Turn on origin shielding + request collapsing.** Pin a single shield POP between all edge POPs and origin so a global cache `MISS` hits origin once, not once-per-POP. Request collapsing (a.k.a. coalescing) dedupes concurrent `MISS`es for the same key into **one** origin fetch while the rest wait — this is your built-in cache-stampede guard at the edge. Confirm both are enabled; they are the difference between a viral event being absorbed vs. a self-inflicted DDoS on origin.
+
+6. **Terminate TLS at the edge and enable HTTP/3.** Edge TLS (TLS 1.3) + `Alt-Svc: h3` / HTTP/3 (QUIC) cuts handshake RTTs and head-of-line blocking on lossy/mobile networks. Keep **origin** connections on TLS too (full/strict, validated cert) — edge-to-origin in cleartext is a classic foot-gun. Force HTTPS with a 308 redirect at the edge; let configure-dns-tls own the cert issuance/HSTS/CAA underneath.
+
+7. **Enable the WAF in detect/log mode first, then enforce.** Order matters — flipping a managed ruleset straight to block will false-positive real traffic and page you at 2am.
+   - Turn on the **managed OWASP Core Rule Set** (SQLi, XSS, RCE, LFI/RFI, protocol anomalies) in **count/log mode**. Watch for 24–72h.
+   - Triage the log: tune or exception the rules that hit legitimate traffic (rich-text editors, `<`/`'` in JSON bodies, signed webhooks). Adjust the **anomaly/paranoia threshold** rather than disabling whole rule families.
+   - **Then flip to block.** Add **custom rules** for your app's known-bad shapes (admin paths from non-office IPs, deprecated API versions, oversized bodies). Example custom rule + edge rate-limit (Cloudflare ruleset-expression syntax):
+
+     ```
+     # block /admin from outside the office, enforce mode
+     (http.request.uri.path matches "^/admin" and not ip.src in $office_ips)  ->  block
+
+     # edge rate-limit /login: >10 req/min/IP -> managed challenge (never reaches origin)
+     when (http.request.uri.path eq "/login")
+       rate_limit { characteristics = ["ip.src"]; period = 60; requests = 10; action = "managed_challenge" }
+     ```
+   - **Edge rate-limiting** for brute-force/scraping (e.g. `/login` > N/min/IP → challenge or 429) — this is the *edge* sibling of origin `rate-limiting`; do coarse volumetric/abuse limits here, fine per-key quotas in the app.
+   - **Bot + DDoS mitigation:** enable managed bot rules + L3/4 and L7 DDoS protection; serve a **JS/managed challenge** (not a hard block) to suspected bots so false positives self-recover. Geo/IP allow/deny rules only where you have a real basis (block sanctioned regions, allow-list admin office IPs) — geo-blocking is blunt and breaks VPN users, so prefer challenges.
+
+## Common Errors
+
+- **Blanket purge on every deploy.** Drops global hit-ratio to zero and stampedes origin each release. Tag with surrogate keys and purge only what changed; let hashed assets expire on their own.
+- **Tracking params in the cache key.** `?utm_source=...` makes every shared link a unique `MISS`. Strip the allow-list's complement before keying.
+- **`Vary: Cookie` (or `User-Agent`).** Fragments the cache to near-uniqueness — effectively no caching. Normalize to a coarse derived dimension (auth-tier, device-class) and `Vary` on that.
+- **One TTL for both browser and edge.** Using `max-age` alone means you can't hold a long edge TTL without baking it into browsers. Separate `s-maxage`/`Surrogate-Control` (edge) from `max-age` (browser).
+- **No `stale-while-revalidate`.** Every expiry becomes a synchronous origin round-trip and a latency spike; add SWR + `stale-if-error`.
+- **No origin shield / no request collapsing.** A cold key fetches origin once *per POP* and concurrent misses each hit origin — a viral object DDoSes you. Enable both.
+- **Cleartext edge-to-origin.** TLS terminates at the edge but the origin pull is HTTP — a MITM goldmine. Use full/strict mode with a validated origin cert.
+- **WAF flipped straight to block.** Guaranteed false positives on launch. Run count/log mode 24–72h, tune, then enforce.
+- **Hard-blocking suspected bots.** A false positive locks out a real user with no recovery. Serve a managed/JS challenge instead so legit clients pass automatically.
+- **Caching `Set-Cookie` / personalized HTML.** Leaks one user's session or data to the next requester. Force `private, no-store` on any response carrying `Set-Cookie` or auth-specific content, and strip `Set-Cookie` from cacheable responses.
+- **Treating the WAF as the fix.** A blocked payload is still an unpatched bug; an attacker who bypasses one rule still wins. Fix the vuln in code (remediate-web-vulnerabilities) — the WAF only buys time.
+
+## Verify
+
+1. **Hit-ratio rises, origin load drops.** Pull the CDN analytics cache hit-ratio before/after; it should climb (static well above 95%, cacheable HTML/API materially up). Confirm origin request rate / bandwidth fell by a corresponding amount — the whole point.
+2. **Edge actually served it.** `curl -sI https://host/path` shows the CDN cache header (`x-cache: HIT`, `cf-cache-status: HIT`, or `x-served-by` with `cache-*-HIT`) and the expected `Cache-Control`/`age`. A second request after a `MISS` must return `HIT`.
+3. **Key normalization works.** Request `?utm_source=x` and the bare URL → both `HIT` the same object (same `age`/etag), proving tracking params are stripped and params are sorted.
+4. **Purge actually invalidates.** Cache an object (`HIT`), change the entity, fire the surrogate-key/path purge, re-request → `MISS` then fresh content. A purge that doesn't flip `HIT`→`MISS` is broken.
+5. **SWR serves stale instantly.** After TTL expiry, the first request returns immediately (stale, background revalidate) rather than blocking on origin; latency stays flat across the expiry boundary.
+6. **TLS + HTTP/3 negotiated.** `curl --http3 -sI https://host` succeeds and `Alt-Svc: h3` is advertised; TLS 1.3 on edge *and* validated TLS on the origin pull (no cleartext hop).
+7. **WAF blocks a test attack.** Send a benign canary payload (`?q=' OR 1=1--`, a reflected-XSS probe) → blocked (403/challenge) with a rule id in the WAF log. Confirm in the same window that real traffic shows ~0 false positives (block-mode log clean of legit requests).
+8. **Edge rate-limit / bot rule fires.** Burst `/login` past the threshold from one IP → challenge/429 at the edge (request never reaches origin logs). A known-good client sails through.
+
+Done = cache hit-ratio is up and measured origin load is down, `HIT`/purge/SWR behave exactly as configured, edge+origin TLS with HTTP/3 are verified, and the WAF blocks a canary attack with zero false positives on real traffic (managed rules in enforce mode, not count).

package/skills/setup-devcontainer-env/SKILL.md

@@ -0,0 +1,131 @@
+---
+name: setup-devcontainer-env
+description: Builds a reproducible Dev Container workspace from a devcontainer.json — pinned base image, language toolchains via devcontainer features, editor config, postCreate provisioning, runtime-injected dev secrets, dependency cache volumes, and parity with the CI base image.
+when_to_use: User wants a clone-and-reopen reproducible workspace, to kill "works on my machine", to onboard a dev fast, or to make a repo Codespaces-ready. Not the production runtime image (dockerfile-optimize), not the multi-service backing stack (compose-local-dev-stack), not host-level version pinning without containers (pin-toolchain-versions).
+---
+
+## When to Use
+
+- "Set up a dev container so everyone gets the same Node/Python/Go and tools"
+- "New hire spends a day installing deps — make it `clone → reopen in container → it just works`"
+- "Make this repo open in GitHub Codespaces"
+- "Our CI uses a different image than our laptops and that's why builds diverge"
+- "I want a disposable, sandboxed workspace I can blow away and rebuild"
+
+NOT this skill:
+- Optimizing the **production runtime** image (multi-stage, distroless, smaller layers) → dockerfile-optimize
+- Standing up **backing services** (Postgres + Redis + a queue) the app talks to → compose-local-dev-stack
+- Pinning tool versions on the **host without containers** (asdf/mise/.tool-versions) → pin-toolchain-versions
+- Secrets in CI/infra, Vault, rotation, leak scanning → secrets-management
+- The CI **workflow** itself (jobs, caching, OIDC) → cicd-pipeline-author
+
+A Dev Container is the **development** environment (editor, debuggers, source bind-mounted, runs as you). Keep it distinct from the lean image you ship.
+
+## Steps
+
+1. **Pick the container source — one of three, decided up front.** Put `.devcontainer/devcontainer.json` at repo root (or `.devcontainer/<name>/devcontainer.json` for multiple).
+
+   | Source key | Use when | Tradeoff |
+   |---|---|---|
+   | `image` | Toolchain is standard, get a workspace in seconds | Customize only via features + postCreate, no custom layers |
+   | `build.dockerfile` | You need OS packages/layers not covered by features, **or want parity with your CI image** | Slower first build; you maintain a Dockerfile |
+   | `dockerComposeFile` + `service` | The dev workspace must join a multi-container stack (db, redis) | Heaviest; stack ownership belongs to compose-local-dev-stack — reference it, don't rebuild it |
+
+   **Default: `build.dockerfile`** pointing at a thin dev Dockerfile that `FROM`s the *same pinned base as CI*. That single decision is what kills environment drift.
+
+2. **Pin the base by digest, never `latest`.** A floating tag means a rebuild three months later silently bumps glibc/openssl and re-breaks "works on my machine". Use the prebuilt devcontainer bases (they ship a non-root `vscode` user + common CLIs) and pin them:
+
+   ```dockerfile
+   # .devcontainer/Dockerfile
+   FROM mcr.microsoft.com/devcontainers/base:ubuntu-24.04@sha256:<digest>
+   # OS packages features can't provide — keep this list tiny
+   RUN apt-get update && apt-get install -y --no-install-recommends \
+         postgresql-client \
+       && rm -rf /var/lib/apt/lists/*
+   ```
+   Refresh the digest deliberately (a dependency-upgrade pass), not by accident.
+
+3. **Install toolchains via `features`, not ad-hoc `apt`/`curl|sh`.** Features are versioned, composable, cache well, and stay portable to Codespaces. Pin each feature to an exact toolchain version — `latest` here reintroduces the drift you just removed.
+
+   ```jsonc
+   {
+     "name": "app-dev",
+     "build": { "dockerfile": "Dockerfile" },
+     "features": {
+       "ghcr.io/devcontainers/features/node:1": { "version": "20.17.0" },
+       "ghcr.io/devcontainers/features/python:1": { "version": "3.12" },
+       "ghcr.io/devcontainers/features/github-cli:1": {}
+     },
+     "remoteUser": "vscode"
+   }
+   ```
+   Keep feature versions in lockstep with `.tool-versions`/CI so host, container, and CI agree. Reserve the Dockerfile for things no feature provides (step 2).
+
+4. **Wire the editor, ports, and lifecycle hooks in the same file.**
+
+   ```jsonc
+   {
+     "customizations": {
+       "vscode": {
+         "extensions": ["dbaeumer.vscode-eslint", "esbenp.prettier-vscode", "ms-python.python"],
+         "settings": { "editor.formatOnSave": true, "terminal.integrated.defaultProfile.linux": "zsh" }
+       }
+     },
+     "forwardPorts": [3000, 5432],
+     "portsAttributes": { "3000": { "label": "web", "onAutoForward": "notify" } },
+     "onCreateCommand": "git config --global --add safe.directory ${containerWorkspaceFolder}",
+     "postCreateCommand": "npm ci && npm run build",
+     "postStartCommand": "npm run db:migrate"
+   }
+   ```
+   - `postCreateCommand` — runs **once** after the container is created. Put idempotent provisioning here: install deps, build, seed. Make it a script (`.devcontainer/post-create.sh`, `set -euo pipefail`) once it grows past one command.
+   - `postStartCommand` — runs **every start**. Cheap/repeatable only (migrations, service warmup). Never put `npm ci` here.
+   - `onCreateCommand` — earliest hook, before secrets/mounts; use for the `safe.directory` fix and base setup.
+
+5. **Run as a non-root user; only `root` when a hook truly needs it.** Set `"remoteUser": "vscode"` so bind-mounted files you create are owned by your host UID, not root. If a feature/base lacks the user, create it in the Dockerfile (`useradd -m vscode`). Need root for one provisioning step? Use `"onCreateCommand"` with `sudo`, then drop back — don't run the whole container as root.
+
+6. **Cache dependencies in named volumes so rebuilds aren't from-scratch.** Bind-mounting the workspace is automatic; the slow part is re-downloading deps. Mount package caches (not `node_modules` inside the workspace — that fights the bind mount):
+
+   ```jsonc
+   "mounts": [
+     "source=app-pnpm-store,target=/home/vscode/.local/share/pnpm/store,type=volume",
+     "source=app-pip-cache,target=/home/vscode/.cache/pip,type=volume"
+   ]
+   ```
+   A fresh `postCreateCommand` then restores from a warm cache instead of the network. For `node_modules`, prefer a volume at the workspace's `node_modules` path so it doesn't sync over the bind mount on macOS/Windows.
+
+7. **Inject dev secrets at runtime — never bake them into the image.** A secret in a Dockerfile layer is in the image history forever and leaks to anyone who pulls it.
+   - Provide a committed `.env.example`; have `postCreateCommand` copy it to a gitignored `.env` the app reads.
+   - For values devs supply, use `"remoteEnv"` referencing host env vars (`"remoteEnv": { "GH_TOKEN": "${localEnv:GH_TOKEN}" }`), or `runArgs: ["--env-file", ".devcontainer/devcontainer.env"]` (gitignored).
+   - In **Codespaces**, real secrets come from repo/Codespaces secrets and appear as env vars — your code must read from env either way, with no host-path assumptions.
+   These are *dev-only* credentials (local DB password, sandbox API key). Real production secrets and rotation belong to secrets-management.
+
+8. **Achieve CI parity by sharing the base, not eyeballing versions.** Point the dev Dockerfile's `FROM` (or a shared build stage) at the **exact pinned image your CI runs on**, and keep feature/toolchain versions equal to CI's. Build the image once and let both consume it; if CI builds its own, diff the resolved versions (`node -v`, `python --version`) in step 9. Parity verified by sameness of inputs beats "looks close".
+
+9. **Verify on a truly clean machine state (see Verify).** The only honest test of reproducibility is a fresh clone + rebuilt container with no host toolchain, including a Codespaces or `--no-cache` rebuild.
+
+## Common Errors
+
+- **`latest`/floating tags on the base image or features.** Looks reproducible until a rebuild months later pulls a new toolchain and breaks the build. Pin the base by `@sha256:` digest and every feature to an exact version.
+- **Installing toolchains with `RUN curl ... | sh` in the Dockerfile.** Unpinned, unportable, and re-downloads on every cache miss. Use a versioned `feature` instead; reserve the Dockerfile for OS packages features can't provide.
+- **Baking secrets/tokens into image layers** (`ENV API_KEY=...` or `COPY .env`). They persist in image history and leak on push. Inject at runtime via `remoteEnv`/`--env-file`/Codespaces secrets; commit only `.env.example`.
+- **Running the container as root.** Files you create on the bind-mounted workspace become root-owned and uneditable from the host. Set `remoteUser` to a non-root user matching your UID.
+- **Heavy work in `postStartCommand`.** It runs on every start, so `npm ci`/full builds make every "reopen" crawl. Heavy/once-only provisioning → `postCreateCommand`; only cheap idempotent steps → `postStartCommand`.
+- **Non-idempotent `postCreateCommand`.** Re-running on rebuild double-seeds the DB or fails on existing rows. Guard with `IF NOT EXISTS`/`--if-not-exists`/existence checks so a rebuild is clean.
+- **`node_modules` (or other native deps) on the host bind mount.** Native binaries built for the host OS break in the Linux container, and sync is slow on macOS/Windows. Put `node_modules` in a named volume at the workspace path.
+- **Dev container drifting from CI.** Different base, different Node minor → the classic green-locally/red-in-CI. Derive both from one pinned base and keep feature versions identical; diff resolved versions in verification.
+- **Testing the rebuild on a machine that still has the host toolchain installed.** It masks "the container forgot to install X" because the host silently provides it. Validate where no host toolchain exists (Codespaces, a clean VM, or a `--no-cache` rebuild).
+- **Editing `devcontainer.json` and expecting a running container to pick it up.** Lifecycle/feature changes only apply on **Rebuild Container**, not reload-window. Always rebuild after config changes.
+
+## Verify
+
+1. **Fresh clone, cold open:** Clone into a directory with no prior build, "Reopen in Container" → container builds, `postCreateCommand` runs to completion (exit 0), and the app builds/starts with **zero host toolchain installed** (uninstall or run in a clean VM).
+2. **Pins are real:** `grep` the Dockerfile and `devcontainer.json` — base is `@sha256:`, every feature has an explicit `version`, no `:latest`. No raw `curl|sh` toolchain installs.
+3. **No baked secrets:** `docker history --no-trunc <image>` shows no tokens/keys/`.env`; the running app reads secrets from env/`.env` injected at runtime. `.env` is gitignored; `.env.example` is committed.
+4. **Non-root + ownership:** `whoami` inside ≠ `root`; a file created in the workspace is owned by your host user, editable from the host.
+5. **CI parity:** Resolved versions inside the container (`node -v`, `python --version`, `go version`) match the CI job's exactly. The dev base and CI base resolve to the same digest (or documented-equal).
+6. **Cache works:** Rebuild the container; `postCreateCommand` restores deps from the cache volume without re-downloading the full dependency set (visibly faster than the first build).
+7. **Idempotent rebuild:** Run "Rebuild Container" twice — second `postCreateCommand` succeeds with no duplicate seeds/errors.
+8. **Codespaces (if targeted):** The repo opens in a Codespace and reaches the same working state with no local-filesystem/host assumptions.
+
+Done = a fresh clone with no host toolchain reaches a building, running app via reopen-in-container; every base/feature is version-pinned; no secret is in any image layer; the container runs non-root; and resolved toolchain versions match CI exactly.