@dsalazardev/ai-hub-cli 0.0.0-stage → 0.1.0

package/skills/shared/scaling/SKILL.md

@@ -0,0 +1,137 @@
+---
+name: scaling
+description: "Use when traffic is growing or about to spike and the system bends under concurrency — deciding what to add and in what order (cache, connection pool, async queue, read replica, more instances) and proving it with a load test against explicit RPS and p95 targets rather than guessing. NOT making one slow request faster or profiling an N+1 (that is `performance`), NOT race-free Redis caches, locks and queue semantics (that is `redis`)."
+tags: [scaling, capacity, load-testing, caching, read-replicas, pgbouncer, k6, devops]
+recommends: [performance, redis, postgresdb, monitoring, deployment, backups, fly-io]
+profiles: []
+origin: ai-hub
+---
+
+# Scaling: survive many requests at once
+
+Performance makes one request faster. Scaling makes many requests survive at the same time. Different problem, different toolbox — don't reach for this one when a single endpoint is slow for a single user (that is `../performance/SKILL.md`).
+
+The whole job in one line: **diagnose the bottleneck, apply the cheapest lever that moves it, re-measure under load.** Repeat until the next bottleneck appears or you hit your target.
+
+**Prime directive: never add infrastructure without a measurement first.** A replica, a queue, or a third app instance you bought on a hunch costs money every month and usually moves the wrong tier. Measure, then add.
+
+## Start here — order of operations
+
+Levers are ordered by payoff per dollar. Caching is nearly free and wins biggest; replicas and autoscaling cost forever. Climb the ladder in order; stop the moment the symptom clears.
+
+| Symptom | Likely bottleneck tier | First lever | Sibling that wires it |
+|---|---|---|---|
+| Slow *only* under load, fine solo | unknown — measure first | USE-method triage (Step 0) | `../monitoring/SKILL.md` |
+| Same reads recomputed for everyone | app/DB doing repeat work | Lever 1 — cache | `../redis/SKILL.md` |
+| `too many clients already` | DB connection slots | Lever 2 — pooler | `../postgresdb/SKILL.md` |
+| Spiky writes time out / drop | synchronous write path | Lever 2 — async queue | `../redis/SKILL.md` |
+| Reads dominate, primary CPU hot | DB read capacity | Lever 3 — read replica | `../postgresdb/SKILL.md` |
+| All tiers healthy, just need throughput | app instance count | Lever 4 — horizontal / autoscale | `../deployment/SKILL.md` |
+
+## Step 0 — find the bottleneck before you add anything
+
+Use the **USE method** (Utilization, Saturation, Errors) on every resource — CPU, memory, disk, network, and the DB connection pool. For each one ask: how busy (U), how much is queued/waiting (S), and any errors (E).
+
+- **Saturation predicts collapse earlier than utilization.** A CPU at 70% util with a growing run-queue is closer to falling over than one at 90% with no queue. Watch queue depth, run-queue length, and connection-pool wait time — those spike before throughput craters.
+- **Read p95/p99, never the mean.** The mean hides the tail your users actually feel; if p50 is 80 ms and p95 is 4 s, a meaningful slice of traffic is having a bad time and the average lies about it.
+- The dashboards, alerts, and SLOs that produce these numbers are `../monitoring/SKILL.md` / observability's job. Scaling *consumes* USE signals; it doesn't build the collectors.
+
+Output of Step 0 is one sentence: "the bottleneck is the DB connection pool / app CPU / origin cache-miss rate." Don't proceed without it.
+
+## Lever 1 — Caching (cheapest, do first)
+
+Cache layers, outermost to innermost — each one removes work the layer behind it would have done:
+
+| Layer | Removes | Typical TTL |
+|---|---|---|
+| CDN / edge | origin round-trip for static + cacheable HTML | minutes–hours |
+| HTTP cache headers (`Cache-Control`, `ETag`) | re-downloads; enables 304s | per-resource |
+| App cache (in-proc / Redis) | recomputed views, serialized payloads | seconds–minutes |
+| Query-result cache | repeated identical DB reads | seconds |
+
+- **Cache the expensive read, not the cheap one.** Caching a 2 ms lookup adds a network hop and an invalidation bug for nothing; cache the 400 ms aggregate everyone hits.
+- **Target >80% hit ratio for general traffic** (static-heavy/CDN routinely hits 95%+). A ratio consistently **<60%** means a broken strategy — wrong cache keys, TTL too low, or churny data that shouldn't be cached.
+- **Beware the cold-cache miss storm.** On deploy or cache flush, every request misses at once and stampedes the origin — the cache that was protecting you now amplifies the load. Mitigating that (stampede protection, single-flight, request coalescing) is cache *correctness* → `../redis/SKILL.md`.
+- If the underlying query is just slow, caching only hides it. Fix the query — N+1, missing index — via `../performance/SKILL.md` before papering over it with TTL.
+
+## Lever 2 — Connection pooling + queues
+
+**Pool DB connections.** Each Postgres connection is a backend process with real memory cost; apps that open a connection per request exhaust `max_connections` fast.
+
+```text
+Bad:  app → opens a fresh DB connection per request → "too many clients already"
+Good: app → PgBouncer (transaction mode) → small pool of reused server connections
+```
+
+- Use **transaction pooling** for stateless web apps: a server connection is held only for the duration of a transaction and released on COMMIT/ROLLBACK — reported real-world effect is roughly 100× effective connection capacity.
+- **Sizing rule:** keep `(number_of_pools × default_pool_size) < max_connections − ~15` (leave headroom for superuser/admin slots). Set `default_pool_size ≈ 1.5–2× vCores` for CPU-bound OLTP — more connections than cores just adds context-switch contention, not throughput.
+- **Gotcha — transaction pooling breaks session-scoped features:** prepared statements (pre-PG14 protocol), `SET` / session GUCs, advisory *session* locks, and `LISTEN/NOTIFY`. Route those to a session-pooling pool or refactor them out. Don't discover this in production.
+
+**Shed spiky writes into a queue.** Queue-based load leveling puts a queue between a bursty producer and a constrained consumer so the consumer drains at its own steady rate; the queue absorbs the spike instead of the synchronous tier melting.
+
+- Move anything that doesn't need a synchronous answer — emails, webhooks, thumbnails, exports — off the request path.
+- Queue *semantics* (race-free rate limits, stalled-job recovery, fencing tokens, durable `SKIP LOCKED` queues) belong to `../redis/SKILL.md` and `../postgresdb/SKILL.md`. Scaling decides *that* you defer work; those decide it's done correctly.
+
+## Lever 3 — Read replicas
+
+Reach for a replica only when Step 0 says **reads dominate and the primary is read-saturated** — not as a reflex.
+
+- **Replicas serve reads, never the source of truth for read-after-write.** Async streaming replication lags; a user who just wrote and immediately reads from a replica may see stale data. Route post-write reads (or that user's whole session for a window) to the primary, or use replica-lag-aware routing.
+- A replica is **not** a write-scaling story and **not**, by itself, an HA/backup story. Writes still all hit one primary; durability and recovery are `../backups/SKILL.md`.
+- Wiring the replication itself — `primary_conninfo`, slots, promotion — is `../postgresdb/SKILL.md`. Scaling decides *add a replica and route reads to it*; postgresdb makes it real.
+
+## Lever 4 — Horizontal scaling & autoscaling
+
+- **Make the app stateless first.** You can't horizontally scale what holds local state — in-memory sessions, sticky uploads on local disk, per-instance caches that must agree. Push session/state to Redis or the DB, then add instances freely.
+- **Autoscale on the saturation metric that actually binds, not CPU alone.** If the real limit is DB connections, scaling app instances on CPU just opens *more* connections and topples the DB faster. Scale on the bottleneck Step 0 found.
+- Choosing a host, rolling deploys, and the autoscaling dials are `../deployment/SKILL.md` plus the platform skill (`../fly-io/SKILL.md`, and siblings for railway/render/vercel). Scaling gives the strategy — how many and triggered by what; the platform gives the knobs.
+
+## Prove it — load testing with k6
+
+Don't claim the system survives. Measure that it does. **k6** (Go core, JS test scripts) reached v1.0.0 on 2025-04-28 under SemVer and is the default OSS load-test tool; the current v1 line is v1.7.x and v2.0.0 shipped in May 2026 (GrafanaCON 2026).
+
+- **Thresholds are the test's pass/fail SLO** — codify the target so the run goes red on its own instead of you eyeballing a graph.
+- **Test a prod-like target, never localhost.** Localhost has no network latency, no real DB, no CDN — it measures your laptop, not your system.
+- Run a ladder, not one shot: **smoke → load → stress → spike → soak** (find the *knee* where latency turns vertical).
+
+```javascript
+import http from 'k6/http';
+import { check, sleep } from 'k6';
+
+export const options = {
+  stages: [
+    { duration: '1m', target: 50 },   // ramp up to 50 virtual users
+    { duration: '3m', target: 50 },   // hold (steady-state load test)
+    { duration: '1m', target: 0 },    // ramp down
+  ],
+  thresholds: {
+    http_req_duration: ['p(95)<500'], // SLO gate: 95% of requests under 500 ms
+    http_req_failed: ['rate<0.01'],   // SLO gate: under 1% errors
+  },
+};
+
+export default function () {
+  const res = http.get(`${__ENV.TARGET_URL}/api/health`);
+  check(res, { 'status is 200': (r) => r.status === 200 });
+  sleep(1);
+}
+```
+
+Full ladder (stage configs for each test type), CI gate snippet, and how to read the summary (p95/p99, `http_req_failed`, spotting the knee) are in **`references/load-testing-k6.md`**.
+
+## Anti-patterns
+
+| Anti-pattern | Why it bites | Do instead |
+|---|---|---|
+| Scaling before measuring | You spend on the wrong tier; symptom persists | Step 0 USE triage, name the bottleneck first |
+| Scaling a stateful app horizontally | Instances disagree; sessions vanish on routing | Make it stateless, externalize state, then scale |
+| Caching cheap work / no TTL strategy | Adds a hop + invalidation bugs for no gain | Cache the expensive read; set deliberate TTLs |
+| Load-testing localhost | Measures your laptop, not production | Test a prod-like target over the network |
+| Reporting mean latency | Hides the tail users actually feel | Gate on p95/p99 |
+| Read replica to absorb writes | Writes still hit one primary; you gain nothing | Replica is read-only; queue/shard writes |
+| DB with no connection pooler | `too many clients already` under any spike | PgBouncer transaction mode + the sizing rule |
+| Autoscaling on CPU while DB connections saturate | More instances = more connections = faster DB death | Autoscale on the binding saturation metric |
+
+## Stop rule & cost
+
+Scale to the *next* bottleneck, then re-measure — don't pre-buy capacity for traffic you don't have. Every lever has a price: caching is ~free, a pooler is cheap, a read replica and autoscaling cost every month and add operational surface. Climb one rung, re-run the load test, and stop when you clear the target. Survived, proven, no further — that's done.

package/skills/shared/scaling/evals/README.md

@@ -0,0 +1,3 @@
+# Evals — scaling
+
+These cases are routing and trigger fixtures, not an automated test harness. `should_trigger` lists prompts the `scaling` description and body must claim (including non-obvious and Catalan/Spanish phrasings); `should_not_trigger` lists near-miss prompts that belong to a real sibling, each tagged with the `route_to` id it should defer to (`performance`, `redis`, `postgresdb`, `monitoring`, `deployment`); `capability` is one launch-readiness scenario with a `must_include` rubric. To run them, read each prompt against `../SKILL.md` and confirm by hand that triggers fire, non-triggers route to the named sibling, and a real answer to the capability scenario covers every rubric line — or feed this file to the repo's eval runner if one is present. The k6 artifacts themselves are checked separately by `../scripts/verify.sh`.

package/skills/shared/scaling/evals/cases.yaml

@@ -0,0 +1,42 @@
+skill: scaling
+
+should_trigger:
+  - prompt: "We're launching on Product Hunt Monday — will our app survive the traffic?"
+    why: "Core spike-survival / capacity question; the canonical reason to reach for scaling."
+  - prompt: "Postgres is throwing `too many clients already` in production."
+    why: "Connection-pool exhaustion under concurrency — the pooler lever (Lever 2), a non-obvious surface that reads like a DB error but is a capacity decision."
+  - prompt: "Should I add caching or a read replica to handle our growth?"
+    why: "Direct lever-choice between Lever 1 and Lever 3; scaling exists to order these."
+  - prompt: "The site is snappy when I test it but crawls once real users hit it at the same time."
+    why: "Non-obvious: sounds like latency but it's a concurrency bottleneck — performance for one user, scaling for many at once."
+  - prompt: "Prepara un test de carga con objetivos de p95 antes del lanzamiento."
+    why: "Spanish — load testing with p95 SLO targets to prove capacity before go-live (k6 + thresholds)."
+  - prompt: "Aguantarem el pic de trànsit del llançament o cau el servidor?"
+    why: "Catalan — explicit traffic-spike survival question, the heart of the skill."
+
+should_not_trigger:
+  - prompt: "This one endpoint takes 3s — find the N+1 and make it fast."
+    route_to: performance
+    why: "Single-request latency / profiling for one user, not concurrency capacity."
+  - prompt: "Make my Redis rate limiter race-free and fix the cache stampede."
+    route_to: redis
+    why: "Cache/queue *correctness* and concurrency semantics, not the whether/where decision scaling makes."
+  - prompt: "Set up streaming replication and the archive_command on Postgres."
+    route_to: postgresdb
+    why: "DB-internal wiring of the replica; scaling decides to add one, postgresdb implements it."
+  - prompt: "Build a Grafana dashboard and p95 alerts for our API."
+    route_to: monitoring
+    why: "Observability / dashboards. Scaling consumes USE signals but doesn't build the collectors."
+  - prompt: "Write the Dockerfile and rolling-deploy config to ship the service."
+    route_to: deployment
+    why: "Packaging and shipping instances; scaling decides how many and when to autoscale, deployment ships them."
+
+capability:
+  - scenario: "Our SaaS expects ~10x traffic next week after a press feature. Stack is a Node API on two small instances behind a load balancer, a single Postgres primary (no pooler), and a CDN for static assets only. Sessions live in app memory. What do we do, in what order, to be ready — and how do we know it'll hold?"
+    must_include:
+      - "Measure first via the USE method (utilization/saturation/errors per tier) and read p95/p99 not the mean before adding anything."
+      - "Caching as the first lever — cache the expensive repeated reads, target >80% hit ratio, and warn about the cold-cache miss storm hitting the origin."
+      - "Put PgBouncer in transaction mode in front of the single primary, with the sizing constraint (pools × default_pool_size < max_connections − ~15; pool ≈ 1.5–2× vCores) — directly addresses the no-pooler risk under 10x."
+      - "Add a read replica ONLY if measurement shows reads dominate and the primary is read-saturated, with the read-after-write/replica-lag caveat (route post-write reads to primary)."
+      - "Make the app stateless (move in-memory sessions out) before scaling instances horizontally, and autoscale on the binding saturation metric, not CPU alone."
+      - "Prove the target with a k6 load test against a prod-like / staging environment (not localhost), gated on thresholds like http_req_duration p(95)<500 and http_req_failed rate<0.01, climbing the smoke→load→stress→spike ladder to find the knee."

package/skills/shared/scaling/references/load-testing-k6.md

@@ -0,0 +1,127 @@
+# Load testing with k6
+
+The full hands-on companion to Lever 4 in `../SKILL.md`. k6 is a load generator: you write a JS test, k6 spins up *virtual users* (VUs) that run it in a loop, and it reports latency distribution, throughput, and error rate against the **thresholds** you declared. k6 hit v1.0.0 on 2025-04-28 (SemVer); the current v1 line is v1.7.x and v2.0.0 shipped May 2026.
+
+## Install
+
+```bash
+brew install k6                  # macOS
+docker run --rm -i grafana/k6 run - <script.js   # containerized, no local install
+# linux: see grafana.com/docs/k6 for the apt/yum repo
+```
+
+Verify with `k6 version`.
+
+## The mental model: VU, iteration, stage
+
+- **VU (virtual user):** one concurrent worker running the default function in a loop. 50 VUs ≈ 50 users hammering at once.
+- **Iteration:** one full pass through `export default function`. `sleep()` inside it models think-time between user actions — without it each VU loops as fast as the machine allows, which is unrealistic.
+- **Stage:** a `{ duration, target }` step that ramps VUs up, holds, or ramps down. Stages chained together shape the load profile.
+
+VUs are *not* requests-per-second. RPS is an emergent result of `VUs × (1 / iteration_time)`. If you need an exact RPS, use the `constant-arrival-rate` executor (below) instead of VU stages.
+
+## Thresholds = the SLO gate
+
+Thresholds turn a test into pass/fail. The process exits non-zero when any threshold is breached — that is what lets CI fail the build.
+
+```javascript
+export const options = {
+  thresholds: {
+    http_req_duration: ['p(95)<500', 'p(99)<1500'], // tail latency budget
+    http_req_failed: ['rate<0.01'],                  // <1% errors
+    http_reqs: ['rate>200'],                         // sustained throughput floor
+  },
+};
+```
+
+Note: in Grafana Cloud, thresholds are evaluated every 60 s, so `abortOnFail` can lag up to a minute — fine for CI gating on the final summary, relevant only if you depend on instant aborts.
+
+## The test-type ladder
+
+Run these in order; each answers a different question. Same script, different `stages`.
+
+```javascript
+// SMOKE — does the script work and the system respond at all? (run this first, always)
+stages: [{ duration: '1m', target: 2 }]
+
+// LOAD — does it hold the expected steady-state traffic within SLO?
+stages: [
+  { duration: '2m', target: 100 },   // ramp
+  { duration: '5m', target: 100 },   // hold at expected peak
+  { duration: '2m', target: 0 },
+]
+
+// STRESS — where is the knee? push past expected until latency goes vertical.
+stages: [
+  { duration: '2m', target: 200 },
+  { duration: '5m', target: 200 },
+  { duration: '2m', target: 400 },
+  { duration: '5m', target: 400 },
+  { duration: '2m', target: 0 },
+]
+
+// SPIKE — survive a sudden surge (Product Hunt / Reddit hug) and recover?
+stages: [
+  { duration: '10s', target: 500 },  // slam
+  { duration: '1m', target: 500 },
+  { duration: '10s', target: 0 },    // does it recover, or stay degraded?
+]
+
+// SOAK — any leak/degradation over hours? memory creep, connection exhaustion.
+stages: [
+  { duration: '5m', target: 100 },
+  { duration: '4h', target: 100 },
+  { duration: '5m', target: 0 },
+]
+```
+
+For exact-RPS targets, swap stages for an arrival-rate executor:
+
+```javascript
+export const options = {
+  scenarios: {
+    steady: {
+      executor: 'constant-arrival-rate',
+      rate: 500, timeUnit: '1s',       // 500 rps regardless of latency
+      duration: '5m',
+      preAllocatedVUs: 100, maxVUs: 800,
+    },
+  },
+};
+```
+
+## Reading the summary
+
+After a run, look at three things in order:
+
+1. **`http_req_failed`** — if errors climbed under load, latency numbers are meaningless. Fix capacity/errors first.
+2. **`http_req_duration` p95/p99** — the tail. p50 looking fine while p99 explodes is the classic "most users fine, some users furious" signal.
+3. **The knee** — the VU/RPS level where p95 stops being flat and turns vertical. That number *is* your capacity ceiling. Everything above it is the danger zone.
+
+Export raw results for graphing:
+
+```bash
+k6 run --out json=results.json script.js
+k6 run --out experimental-prometheus-rw script.js   # → Grafana/Prometheus
+```
+
+## CI gate
+
+k6 exits non-zero on threshold breach, so the gate is just the exit code:
+
+```yaml
+# .github/workflows/load-test.yml (excerpt)
+- name: k6 load test (gates on thresholds)
+  run: k6 run --quiet load.js
+  env:
+    TARGET_URL: ${{ secrets.STAGING_URL }}   # prod-like, never localhost
+```
+
+A failed threshold fails the step, which fails the job. No extra assertion logic needed.
+
+## Rules that keep results honest
+
+- **Target a prod-like environment**, ideally staging that mirrors prod (same instance sizes, same DB tier, same pooler). Localhost measures your laptop.
+- **Ramp gradually** in load/stress tests — instant max load tests cold-start behavior, not steady-state. Use the spike test deliberately when cold-start *is* the question.
+- **Decide cache state on purpose.** Warm the cache first to test steady-state, or test stone-cold to measure the miss-storm against the origin — but know which one you're measuring.
+- **Run from outside your own network** when WAN latency and the CDN matter to the result; a runner inside the same VPC hides the real client experience.

package/skills/shared/scaling/scripts/example.load.js

@@ -0,0 +1,24 @@
+// Example k6 load test shipped with the scaling skill.
+// Used as the checkable artifact for verify.sh and as a copy-paste starting point.
+// Run against a PROD-LIKE target, never localhost: TARGET_URL=https://staging.example.com k6 run example.load.js
+import http from 'k6/http';
+import { check, sleep } from 'k6';
+
+export const options = {
+  stages: [
+    { duration: '1m', target: 50 }, // ramp up to 50 virtual users
+    { duration: '3m', target: 50 }, // hold steady (this is the load test)
+    { duration: '1m', target: 0 },  // ramp down
+  ],
+  thresholds: {
+    http_req_duration: ['p(95)<500'], // SLO: 95% of requests under 500 ms
+    http_req_failed: ['rate<0.01'],   // SLO: under 1% errors
+  },
+};
+
+export default function () {
+  const target = __ENV.TARGET_URL || 'https://staging.example.com';
+  const res = http.get(`${target}/api/health`);
+  check(res, { 'status is 200': (r) => r.status === 200 });
+  sleep(1);
+}

package/skills/shared/scaling/scripts/verify.sh

@@ -0,0 +1,70 @@
+#!/usr/bin/env bash
+# verify.sh — config-validity gate for the scaling skill's k6 artifacts.
+#
+# Read-only by default: it statically asserts that every k6 test script in the
+# skill declares the structure a real load test needs — an `export const options`
+# block, a `thresholds` SLO gate, and a load profile (`stages` or `scenarios`).
+# If `k6` is on PATH it additionally runs `k6 inspect` to confirm the script
+# parses. It never makes a network request and never runs a load test.
+#
+# Exits 0 on a clean/empty target (no k6 scripts found is not a failure).
+# Exits 1 only when a k6 script that DOES exist is missing required structure.
+set -euo pipefail
+
+SKILL_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+fail=0
+checked=0
+
+# Collect candidate k6 scripts: every .js under scripts/. We hard-check real .js
+# files; the reference doc is checked best-effort below. Portable to bash 3.x
+# (no mapfile), so loop over find output via a temp list.
+
+check_script() {
+  local file="$1"
+  local body
+  body="$(cat "$file")"
+  local missing=()
+  grep -q 'export const options' <<<"$body" || missing+=("export const options")
+  grep -q 'thresholds' <<<"$body"            || missing+=("thresholds block")
+  grep -Eq 'stages|scenarios' <<<"$body"     || missing+=("stages/scenarios load profile")
+
+  if [ "${#missing[@]}" -ne 0 ]; then
+    echo "FAIL  $file"
+    printf '      missing: %s\n' "${missing[@]}"
+    fail=1
+  else
+    echo "OK    $file"
+    # Optional deeper validation only when k6 is installed; still no network.
+    if command -v k6 >/dev/null 2>&1; then
+      if k6 inspect "$file" >/dev/null 2>&1; then
+        echo "      k6 inspect: parses cleanly"
+      else
+        echo "      k6 inspect: parse error"
+        fail=1
+      fi
+    fi
+  fi
+  checked=$((checked + 1))
+}
+
+while IFS= read -r f; do
+  [ -n "$f" ] || continue
+  check_script "$f"
+done < <(find "$SKILL_DIR/scripts" -type f -name '*.js' 2>/dev/null || true)
+
+# Best-effort: confirm the reference doc still carries a thresholds example so the
+# prose and the shipped artifact don't drift. Absence is a warning, not a failure.
+ref="$SKILL_DIR/references/load-testing-k6.md"
+if [ -f "$ref" ]; then
+  if grep -q 'thresholds' "$ref"; then
+    echo "OK    $ref (documents thresholds)"
+  else
+    echo "WARN  $ref no longer shows a thresholds example"
+  fi
+fi
+
+if [ "$checked" -eq 0 ]; then
+  echo "No k6 scripts found — nothing to verify (clean)."
+fi
+
+exit "$fail"