@dsalazardev/ai-hub-cli 0.0.0-stage → 0.1.0

package/skills/shared/mongodb/references/transactions-and-ops.md

@@ -0,0 +1,128 @@
+# Transactions & operations (MongoDB 8.2)
+
+## When a transaction is the wrong tool
+
+A single-document write — including an update that touches many fields, arrays, and subdocuments —
+is **already atomic**. You only need a multi-document transaction when two or more documents must
+commit or roll back together (move money between two accounts, flip an order and append a ledger
+entry). If you can restructure so the atomic unit is one document, do that instead: it is faster and
+needs no replica set.
+
+Transactions require a **replica set or sharded cluster**; they do not run on a standalone `mongod`.
+The default has a **60-second** time limit — keep the body short; a long-running transaction holds
+locks and is more likely to abort.
+
+## Retry wrappers
+
+Two distinct retries are mandatory:
+
+- Retry the whole transaction body when the error carries the **`TransientTransactionError`** label.
+- Retry the **commit** when it carries **`UnknownTransactionCommitResult`** (the commit's outcome is
+  unknown; retrying is safe because commit is idempotent).
+
+`withTransaction` does both for you — prefer it.
+
+```javascript
+const session = db.getMongo().startSession();
+try {
+  session.withTransaction(() => {
+    const accts = session.getDatabase("bank").accounts;
+    accts.updateOne({ _id: "A" }, { $inc: { balance: NumberDecimal("-100") } }, { session });
+    accts.updateOne({ _id: "B" }, { $inc: { balance: NumberDecimal("100")  } }, { session });
+  }, { readConcern: { level: "snapshot" }, writeConcern: { w: "majority" } });
+} finally {
+  session.endSession();
+}
+```
+
+If you must hand-roll (no `withTransaction`), the loop is: start → run body → commit; on a
+`TransientTransactionError` label, restart the whole thing; on `UnknownTransactionCommitResult`,
+retry only `commitTransaction()`. Cap retries with a deadline so a persistent error can't spin
+forever.
+
+## Read & write concern semantics
+
+- **`w:"majority"`** — acknowledged once a majority of replica-set members have the write; it cannot
+  be rolled back by a failover. Use it for money and state transitions.
+- **`w:1`** — acknowledged by the primary only; a failover before replication can roll it back. Fine
+  for logs and fire-and-forget.
+- **read concern `"majority"`** — returns data acknowledged by a majority (no dirty reads).
+- **read concern `"snapshot"`** — a consistent point-in-time view; the transaction default.
+- **read concern `"local"`** — the node's latest, possibly un-replicated; fastest, weakest.
+- A **causally consistent session** lets a client read its own writes across operations.
+
+## Atlas tiers
+
+| Tier | For | Limits / features |
+| --- | --- | --- |
+| **M0** | learning, tiny prototypes | free forever, ≤ 5 GB, shared, no SLA; not for real Vector Search |
+| **Flex** (GA Feb 2025) | small prod / spiky load | $8 base, capped at $30/mo, 100 ops/sec (burst 500), 5 GB; supports Atlas Search, Vector Search, Change Streams, Triggers |
+| **M10+** dedicated | production, scale, isolation | from ~$0.08/hr (~$57/mo); dedicated compute, full feature set |
+
+Legacy Serverless / M2 / M5 instances were auto-migrated to Flex. Move off M0 before you depend on
+Search or Vector Search at any real volume.
+
+## Atlas Search & Vector Search
+
+As of MongoDB 8.2, Search and Vector Search ship with Community and Enterprise Server, not just
+Atlas. They are built on a separate search index you define on a collection; queries run through the
+`$search` / `$vectorSearch` aggregation stages. For hybrid (keyword + semantic) ranking, fuse the two
+rankings with `$scoreFusion` (8.2) rather than blending scores in application code.
+
+```javascript
+// Vector similarity search (index defined out of band on the embedding field).
+db.docs.aggregate([
+  { $vectorSearch: {
+      index: "embedding_idx",
+      path: "embedding",
+      queryVector: queryEmbedding,
+      numCandidates: 200,
+      limit: 10
+  }}
+])
+```
+
+## Queryable Encryption
+
+Encrypts fields client-side; the server stores and queries ciphertext without ever seeing
+plaintext. The supported query types have grown:
+
+- **8.0** added **range** queries (`$lt/$lte/$gt/$gte`) on encrypted fields — previously only `$eq`.
+- **8.2** added **prefix, suffix, and substring** query types on encrypted fields.
+
+Use it for regulated/sensitive fields (SSNs, card data). Never store such a value, or any database
+credential, as plaintext.
+
+## Role-based access control
+
+Grant least privilege: a per-application user scoped with the built-in roles (`readWrite` on one
+database, `read` for reporting users) or a custom role, never the cluster-admin user for app
+traffic. The connection string carries the credential — keep it in a secret store, not in the repo.
+
+## Connection-pool knobs
+
+The driver maintains a pool per `MongoClient`; create one client and reuse it for the process
+lifetime (a new client per request exhausts connections — the classic serverless foot-gun).
+
+| Option | What it controls |
+| --- | --- |
+| `maxPoolSize` (default 100) | upper bound on concurrent connections per client |
+| `minPoolSize` | warm connections kept open to avoid cold-start latency |
+| `maxIdleTimeMS` | close a connection idle this long (useful behind a proxy/firewall) |
+| `waitQueueTimeoutMS` | how long an operation waits for a free connection before erroring |
+
+In a serverless/Lambda runtime, cache the client across invocations and keep `maxPoolSize` small —
+many short-lived containers each opening a large pool overruns the server's connection limit.
+
+## Change streams
+
+A resumable feed of collection/database/cluster events, built on the oplog (needs a replica set).
+Filter with a pipeline and persist the `resumeToken` so a restart picks up without gaps.
+
+```javascript
+const cs = db.orders.watch(
+  [{ $match: { operationType: { $in: ["insert", "update"] } } }],
+  { fullDocument: "updateLookup" }   // include the full post-update document
+);
+while (cs.hasNext()) { const change = cs.next(); /* process; save change._id as resume token */ }
+```

package/skills/shared/mongodb/scripts/verify.sh

@@ -0,0 +1,151 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# verify.sh — mongodb skill gate. Run from your PROJECT root.
+#
+# What it does (read-only, idempotent, never writes, never connects to a database):
+#   1. Discovers .js / .mongodb.js files (excluding vendor dirs).
+#   2. Foot-gun scan (advisory [warn] unless noted):
+#        - committed plaintext credential: mongodb://user:pass@ ...      <- the ONLY hard fail
+#        - createIndex(...) with no options object                       <- [warn]
+#        - redundant compound-index prefix (one index is a left prefix of another in the same file)
+#        - $where JS predicate                                           <- [warn]
+#        - unbounded $lookup (a $lookup with no sub-pipeline)            <- [warn]
+#        - allowDiskUse: true (may be masking a missing $match index)    <- [warn]
+#        - money as a JS number in a seed/insert (heuristic)             <- [warn]
+#   3. If `node` is present, runs `node --check` on each file for a syntax pass; else [skip].
+#
+# Exit code: non-zero ONLY when a committed plaintext credential is found. Everything else —
+# missing tools, every heuristic warning, an empty target — is advisory and exits 0.
+#
+# Portability: stock macOS bash 3.2 (no mapfile, no associative arrays). Arrays are initialised
+# so they expand safely under `set -u`.
+
+YELLOW=$'\033[33m'; GREEN=$'\033[32m'; RED=$'\033[31m'; NC=$'\033[0m'
+EXIT=0
+skip() { printf '%s[skip]%s %s\n' "$YELLOW" "$NC" "$*"; }
+note() { printf '%s[warn]%s %s\n' "$YELLOW" "$NC" "$*"; }
+ok()   { printf '%s[ok]%s %s\n'   "$GREEN"  "$NC" "$*"; }
+err()  { printf '%s[fail]%s %s\n' "$RED"    "$NC" "$*"; EXIT=1; }
+
+ROOT="$(pwd)"
+
+# Discover JS / mongosh files. bash 3.2 has no mapfile; read NUL-delimited names into an
+# explicitly-initialised array so expansion is safe under `set -u`.
+JS_FILES=()
+while IFS= read -r -d '' f; do
+  JS_FILES+=("$f")
+done < <(
+  find "$ROOT" \
+    \( -path '*/node_modules/*' -o -path '*/.git/*' -o -path '*/vendor/*' -o -path '*/dist/*' -o -path '*/build/*' \) -prune -o \
+    \( -type f \( -name '*.mongodb.js' -o -name '*.js' \) \) -print0 2>/dev/null
+)
+
+if [ "${#JS_FILES[@]}" -eq 0 ]; then
+  skip "no .js / .mongodb.js files found — nothing to scan"
+  ok "verify.sh passed"
+  exit 0
+fi
+
+# ---- 1. Hard fail: committed plaintext credential in a connection string ----
+# Matches mongodb://user:password@ or mongodb+srv://user:password@ with a literal (non-empty,
+# non-placeholder) password. We treat ${...}, <...>, and process.env as NOT a literal.
+for f in "${JS_FILES[@]}"; do
+  while IFS= read -r line; do
+    case "$line" in
+      *mongodb://*:*@*|*mongodb+srv://*:*@*)
+        # Ignore obvious placeholders / env interpolation.
+        case "$line" in
+          *'${'*|*'<'*'>'*|*process.env*|*':@'*|*'://user:pass@localhost'*) continue ;;
+        esac
+        # Re-confirm there is a colon-delimited secret between scheme and @ that isn't empty.
+        if printf '%s' "$line" | grep -Eq 'mongodb(\+srv)?://[^:/@[:space:]]+:[^@/[:space:]]+@'; then
+          err "$f: plaintext credential in a mongodb connection string — move it to a secret/env var"
+        fi
+        ;;
+    esac
+  done < "$f"
+done
+
+# ---- 2. Advisory foot-gun scan ----
+for f in "${JS_FILES[@]}"; do
+  # createIndex with no options object: createIndex({...})  with no trailing ", { ... }"
+  if grep -Eq 'createIndex\(' "$f" \
+     && ! grep -Eq 'createIndex\([^)]*\},[[:space:]]*\{' "$f"; then
+    note "$f: createIndex(...) with no options object — consider name/partialFilterExpression/unique/expireAfterSeconds"
+  fi
+
+  # $where JS predicate — no index, server-side injection surface.
+  if grep -Eq '\$where' "$f"; then
+    note "$f: \$where runs JS per document (no index, injection surface) — use query operators / \$expr"
+  fi
+
+  # Unbounded $lookup: a $lookup that has no sub-pipeline to project/filter the foreign side.
+  if grep -Eq '\$lookup' "$f" \
+     && ! grep -Eq '\$lookup[^}]*pipeline' "$f"; then
+    note "$f: \$lookup with no sub-pipeline — project/filter the foreign side, or model it away (extended reference)"
+  fi
+
+  # allowDiskUse: true — may be masking a missing $match index.
+  if grep -Eq 'allowDiskUse[[:space:]]*:[[:space:]]*true' "$f"; then
+    note "$f: allowDiskUse:true — confirm via explain it isn't masking a missing \$match index"
+  fi
+
+  # Money as a JS number heuristic: a price/amount/balance/total field assigned a bare number
+  # literal (not NumberDecimal). Catches "amount: 9.99" / "price: 100".
+  if grep -Eiq '(amount|price|balance|total|cost)[[:space:]]*:[[:space:]]*-?[0-9]+(\.[0-9]+)?[[:space:]]*[,}]' "$f" \
+     && ! grep -q 'NumberDecimal' "$f"; then
+    note "$f: money-like field stored as a JS number — use NumberDecimal(\"...\") (Decimal128), floats drift"
+  fi
+done
+
+# Redundant compound-index prefix: within one file, if index keys A are a left prefix of keys B,
+# A is likely redundant. Heuristic on the key-list text between the first {...} of createIndex.
+for f in "${JS_FILES[@]}"; do
+  KEYS=()
+  while IFS= read -r kl; do
+    [ -n "$kl" ] && KEYS+=("$kl")
+  done < <(grep -Eo 'createIndex\([[:space:]]*\{[^}]*\}' "$f" 2>/dev/null \
+            | sed -E 's/.*\{([^}]*)\}.*/\1/' \
+            | tr -d ' ' )
+  n="${#KEYS[@]}"
+  if [ "$n" -ge 2 ]; then
+    i=0
+    while [ "$i" -lt "$n" ]; do
+      j=$((i + 1))
+      while [ "$j" -lt "$n" ]; do
+        a="${KEYS[$i]}"; b="${KEYS[$j]}"
+        # Report once per pair: whichever is the strict left-prefix of the other is redundant.
+        short=""; long=""
+        case "$b" in "$a"*) short="$a"; long="$b" ;; esac
+        case "$a" in "$b"*) short="$b"; long="$a" ;; esac
+        if [ -n "$short" ] && [ "${#short}" -lt "${#long}" ]; then
+          note "$f: index {$short} is a left prefix of {$long} — the longer one likely covers it"
+        fi
+        j=$((j + 1))
+      done
+      i=$((i + 1))
+    done
+  fi
+done
+
+ok "foot-gun scan complete (${#JS_FILES[@]} files)"
+
+# ---- 3. node --check syntax pass ----
+if command -v node >/dev/null 2>&1; then
+  bad=0
+  for f in "${JS_FILES[@]}"; do
+    # mongosh files use shell-only globals (db, ObjectId); node --check only parses syntax, which is fine.
+    if ! node --check "$f" >/dev/null 2>&1; then
+      note "$f: node --check reported a syntax issue (mongosh globals are expected; check real syntax)"
+      bad=$((bad + 1))
+    fi
+  done
+  if [ "$bad" -eq 0 ]; then ok "node --check: all files parse"; fi
+else
+  skip "node not installed — skipping syntax check"
+fi
+
+printf '\n'
+if [ "$EXIT" -eq 0 ]; then ok "verify.sh passed"; else err "verify.sh found a committed credential"; fi
+exit "$EXIT"

package/skills/shared/monitoring/SKILL.md

@@ -0,0 +1,155 @@
+---
+name: monitoring
+description: "Use when setting up uptime and health monitoring, alerts, or on-call basics for a service already in production, so you learn it is down before customers do — health and readiness probes, alerting on SLO error-budget burn rather than raw counts, curing alert fatigue, on-call rotation with escalation, and a status page. NOT instrumenting logs, metrics or traces, or wiring OpenTelemetry (that is `observability`)."
+tags: [monitoring, uptime, alerting, on-call, sre]
+recommends: [observability, deployment, error-handling, domains-dns, scaling]
+origin: ai-hub
+---
+
+# Monitoring
+
+You are wiring up the *outside* view of a service that already shipped: is it alive, is it fast, and when it breaks, does exactly one human get exactly one actionable page. This skill emits a concrete setup — a checker config, a health-endpoint contract, symptom-based alert rules, and an on-call rotation. Not telemetry instrumentation (that is `../observability/SKILL.md`), not the release-gating healthcheck (that is `../deployment/SKILL.md`).
+
+## The one rule
+
+**A page is justified only when there is real user impact AND a human action the system can't take itself.** Everything below descends from this. Internalize the three tiers:
+
+- **Page** (wake someone): users are hurting now and a human must intervene. Checkout returns 5xx. Site unreachable. Error budget burning fast.
+- **Ticket** (look during business hours): degraded but not bleeding. Slow-burn budget use, cert expiring in 14 days.
+- **Dashboard-only** (don't notify): CPU at 80%, a single retry, a transient blip the system already healed.
+
+If an alert doesn't map to an immediate human action, it is not a page — it's noise, and noise trains people to ignore the one page that matters.
+
+## The 4-layer stack
+
+Don't skip layers and don't collapse them — each answers a different question.
+
+| Layer | Answers | Built with | Fires when |
+|---|---|---|---|
+| 1. External uptime probe | "Is it reachable from the outside?" | Uptime Kuma / UptimeRobot / Better Stack | URL down, TLS broken, p95 latency over budget |
+| 2. Health endpoints | "Is the process alive, and are its deps reachable?" | `/livez` + `/readyz` on the service | liveness fails → restart; readiness fails → pull from rotation |
+| 3. SLO burn-rate alert | "Are we spending the error budget too fast?" | Prometheus/Grafana/Better Stack rule | multi-window burn rate exceeds threshold |
+| 4. On-call escalation | "Who acts, and who's the backup?" | PagerDuty / incident.io / Better Stack | a page from layers 1–3 routes + escalates |
+
+Layer 1 catches "the whole thing is gone." Layer 2 catches "a dependency died" before users do. Layer 3 catches "we're degrading faster than we can afford." Layer 4 makes sure a human shows up.
+
+## Pick a tool
+
+Decide by budget, team size, and self-host appetite. Pricing as of 2026-06.
+
+| Tool | Free tier | Check interval | Best for | Paid from |
+|---|---|---|---|---|
+| **Uptime Kuma 2.1.3** | Fully free, self-hosted | down to ~20s | full control, you have a VPS | $0 (you pay the box) |
+| **UptimeRobot** | 50 monitors @ 5-min, 1 status page | 5-min free / 1-min paid | no infra, want managed | $7/mo |
+| **Better Stack** | 10 monitors + incidents + logs | 30s | incident mgmt + on-call in one | $24/mo |
+| **Pingdom** | none | sub-minute | enterprise synthetic + RUM | $15/mo |
+
+Default: **Uptime Kuma** if you already run a VPS (1 GB box is comfortable; needs ~400 MB RAM), **UptimeRobot** free tier if you don't want infra. Kuma 2.1 added Globalping worldwide probe locations (so you test from regions, not just your one box) and built-in domain-expiry monitors. Concrete docker-compose and notification wiring live in `references/tool-setup.md`.
+
+## Health endpoints done right
+
+**Split liveness from readiness — they trigger different machine actions.**
+
+- `/livez` (liveness): is the *process* healthy? If it fails, the orchestrator **restarts** the container. Keep it dumb: just "am I running and not deadlocked." Never check the database here.
+- `/readyz` (readiness): are *dependencies* reachable so I can serve traffic? If it fails, the orchestrator **pulls this instance from the load-balancer rotation** but does not kill it.
+
+**The cheap-probe rule: a probe runs constantly, so it must be <100ms and must not cascade-check every downstream.** Why: if `/livez` pings the DB and the DB is briefly slow, liveness fails, the container restarts, the restart hammers the recovering DB — a restart loop that turns a 30-second blip into an outage.
+
+```python
+# Bad — one /health that cascades and returns 500 on any hiccup.
+# A slow Redis takes the whole service down and triggers restart loops.
+@app.get("/health")
+def health():
+    db.execute("SELECT 1")          # blocks
+    redis.ping()                    # blocks
+    requests.get(PAYMENTS_URL)      # blocks on a third party!
+    return {"status": "ok"}         # 500 if ANY of these throws
+```
+
+```python
+# Good — split, cheap, correct status codes, small JSON.
+@app.get("/livez")                  # liveness: process only. Restart if this fails.
+def livez():
+    return {"status": "alive"}      # 200, ~1ms, touches nothing downstream
+
+@app.get("/readyz")                 # readiness: deps with short timeouts. Pull from LB if this fails.
+def readyz():
+    checks = {"db": ping(db, timeout=0.2), "cache": ping(redis, timeout=0.1)}
+    ok = all(checks.values())
+    return JSONResponse(
+        {"status": "ready" if ok else "degraded", "checks": checks},
+        status_code=200 if ok else 503,   # 503 so the LB pulls this instance
+    )
+```
+
+Do **not** put a third-party API call in readiness — a payment provider's outage shouldn't pull all your instances from rotation and take you fully down. Degrade that path in code (`../error-handling/SKILL.md`), don't fail-closed on it here. Go and FastAPI handler examples are in `references/tool-setup.md`.
+
+## What to actually monitor
+
+The golden checklist. Monitor the *symptom users feel*, not just that one URL returns 200.
+
+- [ ] **Availability** — the critical endpoint(s) reachable, from multiple regions.
+- [ ] **Latency p95/p99** — averages hide the tail; alert on p95/p99 against a budget, not the mean.
+- [ ] **Error rate** — the 5xx-to-total ratio, because raw 5xx count says nothing without traffic volume.
+- [ ] **SSL cert expiry** — alert at 14 days; an expired cert is a full outage that no app metric catches.
+- [ ] **Domain expiry** — alert at 30 days; a lapsed domain is the most embarrassing avoidable outage.
+- [ ] **The critical user journey** — a synthetic check that does login → core action → result, NOT just `GET /`. The homepage can be 200 while checkout is broken.
+
+The homepage being up tells you almost nothing. Probe the path that makes you money.
+
+## Alerts that don't cry wolf
+
+**Alert on symptoms, not causes.** Page on "checkout error rate >2% for 5 min" (user impact), not on "CPU >80%" (a cause that may be harmless and self-resolving). High CPU with happy users is a dashboard line, not a 3am page.
+
+**Use multi-window, multi-burn-rate for SLO alerts** (Google SRE workbook). Burn rate = how fast you're spending the monthly error budget. Require a long *and* a short window to both fire — the long window says "this is real," the short window says "this is still happening," and together they kill false positives from a single spike.
+
+| Burn rate | Window | Budget spent | Severity | Action |
+|---|---|---|---|---|
+| > 14.4 | 1h (+ 5m short) | 2% in 1h | critical | **page** |
+| > 6 | 6h (+ 30m short) | 5% in 6h | warning | **ticket** |
+| > 1 | 3d (+ 6h short) | 10% in 3d | info | review |
+
+Routing: **critical → page**, **warning → ticket**, **info → dashboard/review**. Dedupe and group related alerts into one incident (10 hosts failing the same check = one page, not ten). Set **maintenance windows** so planned deploys don't page anyone. The full burn-rate math and a copy-paste Prometheus-style rule with a `runbook_url` annotation are in `references/burn-rate-and-oncall.md`.
+
+## On-call basics
+
+- **Rotation**: weekly, with a **primary and a secondary**. One person can't be the single point of failure for the system that catches single points of failure.
+- **Escalation policy**: page primary → if no ack in 5–10 min, page secondary → then the manager. No-ack must always hop; an unacked page is a dropped page.
+- **Runbook per alert**: every alert links to a runbook with five fields — Symptom, Impact, First 3 checks, Mitigation, How to escalate. The person paged at 3am should not have to think from scratch. Template in `references/burn-rate-and-oncall.md`.
+- **Status page + comms**: a public status page (Kuma and Better Stack include one) so customers self-serve "is it you or me," cutting inbound during an incident.
+- **Do NOT start a new on-call on Opsgenie** — Atlassian is retiring it (EOL **2027-04-05**; new sales ended 2025-06-04). **Grafana OnCall OSS was also deprecated** (folded into Grafana Cloud IRM). For a new setup use **PagerDuty, incident.io, Better Stack, or Grafana Cloud IRM**.
+
+## Verify it works
+
+**An untested alert is not an alert.** Before you call monitoring "done," prove the wire end-to-end:
+
+1. Trigger a real synthetic failure (stop the service, or point a monitor at a forced-503 route).
+2. Confirm the page actually lands on a phone — not just "the rule exists in the UI."
+3. Let the ack timeout lapse and confirm escalation hops to the secondary.
+4. Restore, and confirm the resolve/all-clear notification fires too.
+
+`scripts/verify.sh` enforces the *structural* half of this on your config: liveness split from readiness, at least one symptom/burn-rate alert with two windows, a runbook reference on every alert, and a banlist for Opsgenie-as-new-setup and homepage-only monitors. Run it in CI so config drift can't silently re-introduce a noisy or untested setup.
+
+## Anti-patterns
+
+| Bad | Why it bites | Do instead |
+|---|---|---|
+| Monitor only the homepage `/` | `/` is 200 while checkout is broken — you learn from customers | synthetic check of the critical journey |
+| Page on CPU/memory threshold | self-resolves, no user impact → alert fatigue | page on the symptom (error rate, latency, availability) |
+| Alert on cause, not symptom | causes are noisy and ambiguous | alert on what the user feels |
+| Alert with no runbook link | the paged human improvises at 3am | every alert links a 5-field runbook |
+| Single-window threshold "by vibes" | one spike pages; tuned by guesswork | multi-window multi-burn-rate against an SLO |
+| Probe cascades every downstream | one slow dep fails the probe → restart loop | cheap <100ms probe, deps in readiness with timeouts |
+| Liveness checks the database | slow DB → restart loop turns a blip into an outage | liveness = process only; DB lives in readiness |
+| Start new on-call on Opsgenie | EOL 2027-04-05, dead end | PagerDuty / incident.io / Better Stack / Grafana Cloud IRM |
+| No secondary on-call | primary asleep/offline → page dropped | primary + secondary + manager escalation |
+| No status page | inbound floods support during incidents | public status page customers can self-serve |
+| Never tested the alert | "the rule exists" ≠ "the page lands" | trigger a synthetic failure, confirm it pages + escalates |
+| Page on warnings | trains people to ignore pages | warning → ticket; only critical → page |
+
+## References
+
+- `references/tool-setup.md` — Uptime Kuma docker-compose (rootless image, volume, port), HTTP + push-heartbeat + SSL/domain-expiry monitors, a synthetic multi-step journey, notification wiring (ntfy / Slack webhook / PagerDuty integration key), a UptimeRobot/Better Stack monitor JSON shape, and Go + FastAPI health-endpoint handlers.
+- `references/burn-rate-and-oncall.md` — full multi-window multi-burn-rate math, a copy-paste Prometheus-style alert rule with `runbook_url`, severity mapping, a sample escalation-policy YAML, and a fill-in runbook template.
+
+Related: `../observability/SKILL.md` (what the app emits), `../deployment/SKILL.md` (release-gating healthcheck + rollback), `../error-handling/SKILL.md` (degrade in code), `../domains-dns/SKILL.md` (provision certs/DNS), `../scaling/SKILL.md` (survive the load monitoring detected).

package/skills/shared/monitoring/evals/README.md

@@ -0,0 +1,3 @@
+# Evals — monitoring
+
+These cases are run by the repo's eval harness, which loads this skill's `SKILL.md` (description + body) and scores three things with a judge model: that each `should_trigger` prompt selects `monitoring`, that each `should_not_trigger` prompt routes to the named sibling instead (every `route_to` id — `observability`, `error-handling`, `deployment`, `scaling`, `domains-dns` — is a real skill in this catalog), and that a response to the `capability` scenario covers the `must_include` rubric. The rubric is graded on substance, not exact wording: a setup that splits liveness from readiness, picks a concrete dated tool, monitors the critical journey plus cert/domain expiry, alerts on a two-window symptom/burn-rate rule, wires escalation + runbooks, says to test the page end-to-end, and steers clear of Opsgenie for a new setup should pass even if phrased differently. Run them with the standard harness against this directory; there is no special setup beyond the catalog being present.

package/skills/shared/monitoring/evals/cases.yaml

@@ -0,0 +1,47 @@
+skill: monitoring
+
+should_trigger:
+  - prompt: "Set up uptime monitoring and alert me if the site goes down."
+    why: Core uptime trigger — the external probe layer of the stack.
+  - prompt: "Add /healthz and /readyz endpoints and have something probe them."
+    why: Health-endpoint contract; liveness vs readiness split is core to this skill.
+  - prompt: "We get paged at 3am for CPU spikes that self-resolve — fix our alerting."
+    why: Non-obvious. Alert fatigue / symptom-based tuning; the cause-not-symptom anti-pattern.
+  - prompt: "Create an on-call rotation with escalation and a public status page."
+    why: On-call basics — rotation, escalation policy, status page.
+  - prompt: "Alert on our SLO error-budget burn rate instead of raw error count."
+    why: Non-obvious. Multi-window multi-burn-rate alerting is a named technique here.
+  - prompt: "An untested alert isn't an alert — how do I prove my pages actually land?"
+    why: Non-obvious. The verify-it-works section; trigger synthetic failure, confirm page + escalation.
+  - prompt: "Monta monitorització d'uptime i que m'avisi si l'API torna 5xx."
+    why: Catalan trigger for uptime + symptom-based 5xx alerting.
+
+should_not_trigger:
+  - prompt: "Add OpenTelemetry tracing and a metrics dashboard for span latency."
+    route_to: observability
+    why: Instrumentation/telemetry — the inside view of what the app emits, not whether it's alive.
+  - prompt: "Add retry and a circuit breaker so failed calls degrade gracefully."
+    route_to: error-handling
+    why: Handling the failure in code, not detecting and escalating it operationally.
+  - prompt: "Add a HEALTHCHECK to my Dockerfile and a rollback step in CI."
+    route_to: deployment
+    why: Release-gating healthcheck + rollback, not the continuous external probe after release.
+  - prompt: "Our app falls over at 10x traffic — set up autoscaling and load shedding."
+    route_to: scaling
+    why: Capacity/survival, not detection of failure.
+  - prompt: "Renew and provision the SSL cert and fix the DNS records for the domain."
+    route_to: domains-dns
+    why: Cert/DNS provisioning; monitoring only watches expiry, it doesn't provision.
+
+capability:
+  - scenario: >
+      I have a FastAPI service on a Hetzner VPS behind Cloudflare. Set up monitoring,
+      alerts, and on-call so I find out before customers do.
+    must_include:
+      - Splits liveness (/livez, restart-if-fail) from readiness (/readyz, pull-from-rotation-if-fail) and states the cheap-probe rule (<100ms, don't cascade every downstream).
+      - Picks a concrete tool with dated rationale (e.g. Uptime Kuma self-host on the VPS vs UptimeRobot if no infra), not a vague "use a monitor."
+      - Monitors the critical user journey via a synthetic check plus cert and domain expiry, not just GET /.
+      - Defines a symptom/SLO alert with two windows (burn-rate, long + short), not a bare CPU/memory threshold.
+      - Sets up an escalation policy (primary + secondary + manager, ack timeout) and links every alert to a runbook.
+      - Says to test the alert end-to-end (trigger a synthetic failure, confirm the page lands and escalation hops).
+      - Avoids recommending Opsgenie for a new setup (EOL 2027-04-05); suggests PagerDuty / incident.io / Better Stack / Grafana Cloud IRM instead.

package/skills/shared/monitoring/references/burn-rate-and-oncall.md

@@ -0,0 +1,128 @@
+# Burn-rate alerting and on-call
+
+The full math behind the SLO table in SKILL.md, plus copy-paste alert rules, escalation, and a runbook template.
+
+## What "burn rate" means
+
+Pick an SLO, e.g. **99.9% availability over 30 days**. The **error budget** is the allowed bad fraction: `1 - 0.999 = 0.1%` of requests over the month.
+
+**Burn rate = how many times faster than "even" you are spending that budget.**
+
+- Burn rate `1` = you'd exactly exhaust the 30-day budget in 30 days. Fine.
+- Burn rate `14.4` = you'd exhaust the *entire month's* budget in `30d / 14.4 ≈ 2h`. Page now.
+
+Why 14.4: it is the rate that burns **2% of the monthly budget in 1 hour** — the Google SRE workbook's recommended fast-burn page threshold. The other rows: **6×** burns ~5% in 6h (ticket), **1×** burns ~10% in 3d (slow review).
+
+## Why two windows
+
+A single window either pages on a momentary spike (short window alone) or reacts too slowly (long window alone). **Require a long AND a short window to both exceed the burn threshold:**
+
+- The **long window** confirms the burn is sustained ("this is real").
+- The **short window** confirms it's still happening right now ("don't page me for an outage that already ended"). It also gives fast reset once you recover.
+
+This combination is what kills false positives.
+
+## Prometheus-style alert rule
+
+Symptom-based, two windows, with a `runbook_url` annotation. Adjust `job`/`code` to your metric names.
+
+```yaml
+groups:
+  - name: slo-burn-rate
+    rules:
+      # CRITICAL — page. Fast burn: 14.4x over 1h, confirmed by a 5m short window.
+      - alert: CheckoutErrorBudgetFastBurn
+        expr: |
+          (
+            sum(rate(http_requests_total{job="checkout",code=~"5.."}[1h]))
+            / sum(rate(http_requests_total{job="checkout"}[1h]))
+          ) > (14.4 * 0.001)
+          and
+          (
+            sum(rate(http_requests_total{job="checkout",code=~"5.."}[5m]))
+            / sum(rate(http_requests_total{job="checkout"}[5m]))
+          ) > (14.4 * 0.001)
+        labels:
+          severity: critical        # routes to page
+        annotations:
+          summary: "Checkout burning error budget 14.4x (1h+5m)"
+          runbook_url: "https://wiki.example.com/runbooks/checkout-5xx"
+
+      # WARNING — ticket. Slower burn: 6x over 6h, confirmed by a 30m short window.
+      - alert: CheckoutErrorBudgetSlowBurn
+        expr: |
+          (
+            sum(rate(http_requests_total{job="checkout",code=~"5.."}[6h]))
+            / sum(rate(http_requests_total{job="checkout"}[6h]))
+          ) > (6 * 0.001)
+          and
+          (
+            sum(rate(http_requests_total{job="checkout",code=~"5.."}[30m]))
+            / sum(rate(http_requests_total{job="checkout"}[30m]))
+          ) > (6 * 0.001)
+        labels:
+          severity: warning          # routes to ticket
+        annotations:
+          summary: "Checkout slow error-budget burn (6h+30m)"
+          runbook_url: "https://wiki.example.com/runbooks/checkout-5xx"
+```
+
+Note: every alert carries `runbook_url`, and the rule is a *ratio* (symptom), never a bare `cpu`/`memory` threshold.
+
+## Severity → routing
+
+| Label | Where it goes | Who acts | When |
+|---|---|---|---|
+| `critical` | PagerDuty/Better Stack page → escalation policy | on-call, now | immediately, any hour |
+| `warning` | ticket queue / Slack channel | team | next business hours |
+| `info` | dashboard / weekly review | nobody paged | retro |
+
+## Escalation policy (sample)
+
+```yaml
+# Vendor-agnostic shape — maps to PagerDuty / incident.io / Better Stack.
+escalation_policy:
+  name: checkout-oncall
+  on_call:
+    primary:   "@oncall-primary"     # weekly rotation
+    secondary: "@oncall-secondary"   # weekly rotation, offset
+  steps:
+    - notify: primary
+      ack_timeout_minutes: 10        # no ack -> escalate
+    - notify: secondary
+      ack_timeout_minutes: 10
+    - notify: "@eng-manager"
+  repeat_until_acked: true
+  resolve_notifies: true             # send the all-clear too
+# Do NOT build this on Opsgenie (EOL 2027-04-05) or Grafana OnCall OSS (deprecated).
+```
+
+## Runbook template
+
+Every alert's `runbook_url` points to one of these. Fill it in before the alert ships, not at 3am.
+
+```markdown
+# Runbook: <alert name>
+
+## Symptom
+What the alert literally measures (e.g. "checkout 5xx ratio > 14.4x SLO burn, 1h+5m").
+
+## Impact
+What the user experiences (e.g. "checkout failing → lost revenue, ~X orders/min").
+
+## First 3 checks
+1. Dashboard link: <url> — is it errors or latency?
+2. Recent deploys: <link> — did we ship in the last 30 min? Consider rollback (see ../deployment/SKILL.md).
+3. Dependency status: DB / cache / payment provider status pages.
+
+## Mitigation
+- Fastest safe action (rollback / feature-flag off / scale out — see ../scaling/SKILL.md).
+- Holding action if root cause unknown (drain region, enable degraded mode).
+
+## Escalation
+- If not mitigated in 15 min, page secondary and open an incident.
+- Post to the public status page; notify #incidents.
+
+## Dashboards
+- <links to the relevant observability dashboards — see ../observability/SKILL.md>
+```