sanook-cli 0.4.0

package/skills/incident-response-sre/SKILL.md

@@ -0,0 +1,78 @@
+---
+name: incident-response-sre
+description: Drives live incident response and postmortems SRE-style: severity triage (P0–P3), log/metric/trace correlation to find what changed, safe mitigation, comms updates, and blameless postmortem with action items. Triggers during an active incident/outage, on-call triage, or writing a postmortem afterward.
+when_to_use: incident/outage เกิดอยู่, on-call ต้อง triage, service degraded, หรือเขียน postmortem หลังเหตุ
+---
+
+## When to Use
+
+- An active incident is happening: errors spiking, latency up, partial/full outage, data not flowing, customers reporting breakage.
+- On-call triage: an alert fired and you must decide severity + whether to page humans.
+- A service is degraded but not down (elevated error rate, slow tail latency, queue backlog growing).
+- After the incident is resolved and you need to write a blameless postmortem.
+
+Do NOT use for routine bug fixing with no live blast radius, planned maintenance, or feature work — use normal debugging instead.
+
+## Steps
+
+**Phase A — Stabilize first, diagnose second (mitigation before root cause).**
+
+1. **Declare + triage severity.** Pick one and state it explicitly with scope:
+   - **P0** — full outage / data loss / security breach / money-affecting. Mitigate now, page on-call.
+   - **P1** — major feature down or broad degradation, no full workaround.
+   - **P2** — partial degradation, workaround exists, limited users.
+   - **P3** — minor / cosmetic / single-user, no urgency.
+   Write impact in user terms: *what* is broken, *who/how many* affected, *since when*. If you can't quantify yet, say "scope unknown, treating as P1 until proven smaller" — never downgrade on a guess.
+
+2. **Build the "what changed" timeline.** Most incidents are caused by a recent change. List, with timestamps, the last 24–48h of:
+   - deploys / releases / rollbacks (check CI/CD history, git log, image tags)
+   - config / feature-flag / env-var changes
+   - infra changes (scaling, DB migration, cert rotation, DNS, dependency upgrade)
+   - traffic shifts (spike, new client, retry storm, upstream outage)
+   Overlay the incident start time against this. The change immediately preceding the symptom onset is your prime suspect.
+
+3. **Correlate signals — logs + metrics + traces together, not in isolation.**
+   - **Metrics** tell you *where/when* (which service, which endpoint, error-rate vs latency vs saturation — the golden signals). Find the first metric that broke and its exact start time.
+   - **Logs** tell you *what* (the actual error string, stack trace, status code). Grep the failing service around the metric break time; read the FIRST errors, not the loudest.
+   - **Traces** tell you *which hop* in the request path failed (DB? upstream dep? timeout? auth?).
+   Pin all three to the same time window. A latency spike + DB connection-pool-exhausted logs + traces stalling at the DB call = one coherent story.
+
+4. **Hypothesis-driven correlation — don't oversimplify.** Form 2–3 candidate root causes from the timeline + signals. For each, state the one observation that would confirm or kill it, then check that observation. Beware: correlation ≠ cause (a deploy at the same time may be innocent); the loudest error may be a downstream *symptom*, not the source. Keep going until the signal chain is coherent end-to-end — stop only when remaining hypotheses are killed.
+
+5. **Mitigate with the safest reversible action FIRST — before any permanent fix.** Prefer, in order: roll back the suspect deploy/config → disable the suspect feature flag → scale up / add capacity → shed load / rate-limit / shut a non-critical path → failover. Reversible mitigation is allowed even before root cause is 100% confirmed if it's safe and likely to help. **Never** apply an irreversible action (drop/delete data, force-push, destructive migration) as a mitigation — escalate to a human first. Confirm recovery via the same metric that broke in step 3, not by assumption.
+
+6. **Post status comms.** Short, factual, no blame, on a cadence (e.g. every 30 min while P0/P1). Each update: current impact → what's being done → next update time. On resolution: "mitigated/resolved at <time>, root cause <known/under investigation>, monitoring." State unknowns plainly — don't speculate publicly.
+
+**Phase B — After recovery: blameless postmortem.**
+
+7. **Write the postmortem** (only after the incident is mitigated/closed). Required sections:
+   - **Summary** — one paragraph: what broke, impact, duration.
+   - **Timeline** — UTC timestamps from detection → mitigation → resolution, including key diagnostic steps.
+   - **Impact** — quantified (users, requests, $, SLO/error-budget burn).
+   - **Root cause + contributing factors** — the trigger AND the conditions that let it become an incident (missing alert, no rollback path, silent dependency). Multiple factors usually, not one.
+   - **What went well / what was slow** — detection time, mitigation time, gaps in tooling/visibility.
+   - **Action items** — each is concrete, *owned*, *dated*, and prioritized (P0/P1...). Distinguish "prevent recurrence" from "detect faster" from "mitigate faster". No vague "be more careful".
+
+8. **Blameless language.** Attack the system, never the person. "The deploy step had no automated canary check" not "X deployed bad code". People act reasonably given the info they had; if a human could cause an outage with a normal action, the *system* lacked a guardrail.
+
+9. **Feed back into a runbook.** If this incident class can recur, capture the detection signal → mitigation steps → verification as a reusable runbook so next time is faster. Convert the highest-value action item into a guardrail (alert, canary, rollback automation, flag).
+
+## Common Errors
+
+- **Chasing root cause while the system burns.** Mitigation (rollback/flag-off) comes before diagnosis. A reverted deploy that buys 30 min is worth more than the perfect RCA mid-outage.
+- **Trusting the loudest error.** The error filling the logs is often a downstream *symptom* (cascading timeouts, retry amplification). Trace upstream to the first failure in time, not the most frequent.
+- **"Deploy at the same time = the cause."** Temporal coincidence is a hypothesis, not proof. A config change, traffic spike, or expiring cert can hide behind an innocent deploy. Verify the causal link.
+- **Premature severity downgrade.** "Probably just a few users" → it's the whole region. When scope is unknown, hold the higher severity until you've *measured* it smaller.
+- **Irreversible "fix" under pressure.** Deleting rows / dropping a table / force-pushing / running a destructive migration to "clear the bad state" turns a recoverable outage into permanent data loss. Escalate to a human before anything irreversible.
+- **Looking at one signal only.** Metrics-only = you see the *symptom* but not *why*. Logs-only = you miss the blast radius. Traces-only = you miss the trend. The story lives at the intersection, pinned to one time window.
+- **Blameful postmortem.** Naming who-did-it makes people hide info next time and kills the learning. The action item is a missing guardrail, never a missing person.
+- **Action items with no owner/date.** "Improve monitoring" never ships. Each item needs a name, a date, and a priority, or it's decoration.
+- **Declaring "fixed" without watching recovery.** Mitigation applied ≠ recovered. Confirm on the exact metric that originally broke before standing down.
+
+## Verify
+
+- **During incident:** the metric that first broke (step 3) is back to baseline AND held there for a sustained window — not a momentary dip. Error rate / latency / saturation confirmed normal on the dashboard, not assumed.
+- **Mitigation safety:** every action taken was reversible, or a human explicitly approved any irreversible one. You can state how to undo each.
+- **Severity correct in hindsight:** measured impact matches (or was higher than) the declared severity — you didn't under-call it.
+- **Postmortem complete:** all required sections present; root cause has ≥1 contributing factor beyond the trigger; every action item has owner + date + priority; language is blameless (no person named as cause).
+- **Loop closed:** at least one action item became a real guardrail (alert/canary/rollback/flag) or a runbook so the same incident is faster/prevented next time.

package/skills/k8s-debug-workload/SKILL.md

@@ -0,0 +1,135 @@
+---
+name: k8s-debug-workload
+description: Systematically diagnoses live Kubernetes workload failures — CrashLoopBackOff, ImagePullBackOff, OOMKilled, pending pods, failing probes — by gathering describe/logs/events/node status and isolating root cause. Triggers when a pod won't start, keeps restarting, or a deployment is stuck/unhealthy in a cluster.
+when_to_use: pod CrashLoopBackOff/ImagePullBackOff/OOMKilled/Pending, probe fail, rollout ค้าง, service ไม่มา endpoint
+---
+
+## When to Use
+
+A live K8s workload is broken and you have `kubectl` access. Specifically:
+
+- Pod stuck `Pending`, `ContainerCreating`, `Init:*`, or `Terminating`
+- Pod `CrashLoopBackOff`, `Error`, `OOMKilled`, or restart count climbing
+- `ImagePullBackOff` / `ErrImagePull` / `InvalidImageName`
+- Readiness/liveness/startup probe failing — pod `Running` but `0/1 READY`
+- `kubectl rollout` stuck, deployment shows old + new ReplicaSet both alive
+- Service has no endpoints / traffic 503s even though pods look up
+
+Do NOT use for: writing new manifests from scratch, Helm chart authoring, cluster provisioning. This skill is for diagnosing something already deployed.
+
+## Steps
+
+**0. Pin the target. Never debug "the cluster" — debug one object.**
+```
+kubectl get pods -n <ns> -o wide          # find the bad pod, note NODE + IP
+kubectl get deploy,rs,pod -n <ns> -l app=<x>   # see the ownership chain
+```
+Always pass `-n <ns>` explicitly. Default namespace is a trap. Grab the exact pod name (`<deploy>-<rs-hash>-<rand>`) for everything below.
+
+**1. Read the symptom from `STATUS` + `RESTARTS`, then branch. Do not guess — the status string already names the failure layer:**
+
+| STATUS | Layer | Jump to |
+|---|---|---|
+| `Pending` | scheduling / quota | Step 4 |
+| `ContainerCreating` stuck | volume / CNI / image | Step 2 + Step 4 |
+| `ImagePullBackOff`/`ErrImagePull` | image / registry auth | Step 2 |
+| `CrashLoopBackOff`/`Error` | container exits | Step 3 |
+| `OOMKilled` (in RESTARTS detail) | memory limit | Step 3 + Step 4 |
+| `Running` but `0/N READY` | probe / app slow-start | Step 5 |
+| Pod fine, Service 503 | endpoints / selector / port | Step 6 |
+
+**2. Always start with `describe` — events are the highest-signal source and most people skip them.**
+```
+kubectl describe pod <pod> -n <ns>
+```
+Read the bottom `Events:` block first. It literally prints `Failed to pull image`, `0/3 nodes available: insufficient memory`, `Liveness probe failed: ...`. Then for image pulls:
+- `ErrImagePull` + `not found` → wrong tag/repo. Check `Image:` field vs what was actually pushed.
+- `ErrImagePull` + `unauthorized`/`denied` → missing/expired `imagePullSecrets`. Verify: `kubectl get sa <sa> -n <ns> -o jsonpath='{.imagePullSecrets}'` and that the secret exists + is type `kubernetes.io/dockerconfigjson`.
+- `InvalidImageName` → typo / bad chars in image string.
+- Stuck `ContainerCreating` with no pull error → volume mount or CNI, go to Step 4.
+
+**3. For crashes, get BOTH current and previous logs — the crash output lives in `--previous`.**
+```
+kubectl logs <pod> -n <ns> --previous --tail=200       # the run that died
+kubectl logs <pod> -n <ns> --tail=100                  # current attempt
+kubectl logs <pod> -n <ns> -c <container> --previous   # if multi-container/init
+```
+Then decode the exit:
+```
+kubectl get pod <pod> -n <ns> -o jsonpath='{.status.containerStatuses[*].lastState.terminated}'
+```
+Read `reason` + `exitCode`:
+- `OOMKilled` → real memory cap hit. Go to Step 4 for limits. Bump `resources.limits.memory` or fix the leak — do NOT just raise the limit blindly if RSS grows unbounded.
+- exit `137` → SIGKILL (OOM or failed liveness kill). Cross-check `Events` for `Liveness probe failed`.
+- exit `143` → SIGTERM, usually probe-triggered restart or graceful shutdown loop.
+- exit `1`/`2` + app stack trace in `--previous` → app bug / bad config. Check env vars + mounted ConfigMap/Secret are present: `kubectl exec <pod> -- env` won't work if crashing, so read the manifest's `envFrom`/`valueFrom` and confirm the referenced ConfigMap/Secret exists.
+- exit `0` looping → app runs to completion then restarts; wrong `restartPolicy` or missing long-running command.
+- CrashLoop with empty logs → fails before logging; check `command`/`args` override, entrypoint, or missing config file mount.
+
+**4. For `Pending`/OOM, check scheduling constraints and node pressure — the scheduler tells you exactly why in events.**
+```
+kubectl describe pod <pod> -n <ns> | grep -A10 Events    # "0/N nodes available: ..."
+kubectl describe node <node>                             # Allocatable, Taints, pressure
+kubectl top pod <pod> -n <ns>; kubectl top nodes         # needs metrics-server
+```
+Match the scheduler reason:
+- `insufficient cpu/memory` → requests too high vs `Allocatable`, or nodes full. Lower `requests` or scale nodes.
+- `node(s) had untolerated taint` → pod lacks matching `tolerations`. Check `Taints:` on node.
+- `didn't match node affinity/selector` → `nodeSelector`/`affinity` points at labels no node has.
+- `had volume node affinity conflict` → PV is zone-locked, pod scheduled to wrong zone.
+- `pod has unbound immediate PersistentVolumeClaims` → PVC `Pending`: `kubectl get pvc -n <ns>` then `describe pvc` (usually no provisioner / no matching StorageClass).
+- Node shows `MemoryPressure`/`DiskPressure True` → node-level, not pod-level; evictions incoming.
+
+**5. For `Running` but not `READY`, the probe config is wrong far more often than the app.**
+```
+kubectl describe pod <pod> -n <ns> | grep -iA3 -e Liveness -e Readiness -e Startup
+kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.containers[*].readinessProbe}'
+```
+Check, in order:
+- Probe `path`/`port` actually served by the app? Wrong port = permanent fail. Confirm against `containerPort` and what the app binds.
+- `initialDelaySeconds` too short for slow boot → use a `startupProbe` instead of inflating liveness delay.
+- Probe hits a path requiring auth → returns 401/403, counts as fail; use a dedicated unauthenticated `/healthz`.
+- From inside (if it stays up): `kubectl exec <pod> -n <ns> -- wget -qO- localhost:<port><path>` to reproduce what kubelet sees.
+- Liveness too aggressive → kills a healthy-but-busy pod, looks like CrashLoop (exit 137/143). Relax `timeoutSeconds`/`failureThreshold`.
+
+**6. For "pod up but Service dead", verify the selector→endpoint chain — a 503 with healthy pods is almost always a selector or port mismatch.**
+```
+kubectl get endpointslices -n <ns> -l kubernetes.io/service-name=<svc>   # EMPTY = broken
+kubectl get svc <svc> -n <ns> -o wide
+kubectl describe svc <svc> -n <ns>
+```
+- Empty endpoints → Service `selector` labels don't match pod labels (`kubectl get pod --show-labels`), OR pods aren't `READY` (go to Step 5 — unready pods are excluded from endpoints).
+- Endpoints present but traffic fails → Service `targetPort` ≠ container's listening port. Confirm `targetPort` maps to the real `containerPort`/app port.
+- DNS: `kubectl run tmp --rm -it --image=busybox -n <ns> -- nslookup <svc>` to confirm resolution; check `kube-dns`/CoreDNS pods are up if it fails.
+- `NetworkPolicy` silently dropping traffic: `kubectl get netpol -n <ns>` — a default-deny with no matching allow rule blackholes connections with no error.
+
+**7. Propose ONE narrow fix, name the exact field changed, and give a verify command. Don't shotgun multiple changes at once — you won't know which one worked.**
+
+## Common Errors
+
+- **Skipping `--previous`** — `kubectl logs <pod>` on a crashlooping pod shows the *current* (already-restarted, possibly empty) attempt. The actual crash output is in `--previous`. This is the #1 missed clue.
+- **Trusting `kubectl logs` when the pod never started** — Init containers and pre-entrypoint failures produce no app logs; the answer is in `describe` Events, not logs.
+- **Raising the memory limit to "fix" OOMKilled** — if RSS grows unbounded it's a leak; a higher limit just delays the kill and masks the bug. Confirm steady-state vs leak via `kubectl top pod` over time before touching limits.
+- **Confusing liveness vs readiness** — a failing *readiness* probe takes the pod out of Service endpoints (traffic stops) but does NOT restart it. A failing *liveness* probe *restarts* it (exit 137/143, looks like a crash). The fix differs: readiness = app not ready / wrong probe; liveness = probe too aggressive or app hung.
+- **Forgetting unready pods are excluded from endpoints** — a "Service has no endpoints" bug is frequently a probe problem in disguise (Step 5), not a selector problem.
+- **`top` returns error** — needs metrics-server installed; if absent, read `resources` requests/limits from the manifest and node `Allocatable` from `describe node` instead.
+- **Editing the live pod instead of the controller** — `kubectl edit pod` changes get wiped on the next restart. Patch the Deployment/StatefulSet (`kubectl set resources` / `kubectl edit deploy`) so the change survives.
+- **`ContainerCreating` blamed on the image** when it's actually a stuck PVC or CNI — check Events for `FailedMount`/`FailedAttachVolume` vs pull errors; they look similar from the outside.
+- **Ignoring the ownership chain** — restarting a single pod when the Deployment template is the bug just respawns the same broken pod. Fix the template, then `rollout restart`.
+
+## Verify
+
+After applying a fix, confirm with a runnable check — never declare done on "looks fixed":
+
+```
+kubectl rollout status deploy/<name> -n <ns> --timeout=120s   # waits, exits non-zero on fail
+kubectl get pods -n <ns> -l app=<x> -w                        # watch RESTARTS stay 0, READY = N/N
+```
+Then re-confirm the original symptom is gone:
+- Crash fixed → `RESTARTS` stops climbing for ≥2 min and `lastState.terminated` is no longer present.
+- Image fixed → STATUS leaves `ImagePullBackOff`, reaches `Running`.
+- Pending fixed → pod has a `NODE` assigned in `-o wide`.
+- Probe fixed → `READY` shows `N/N`, no new `probe failed` events.
+- Service fixed → `kubectl get endpointslices -n <ns> -l kubernetes.io/service-name=<svc>` lists pod IPs, and a request from an in-cluster `busybox` pod succeeds.
+
+If the verify command fails, do NOT relax the probe/assertion or bump a limit to force green — go back to the matching step and find the real cause.

package/skills/k8s-manifest-review/SKILL.md

@@ -0,0 +1,86 @@
+---
+name: k8s-manifest-review
+description: Reviews and writes Kubernetes / Helm manifests for production-readiness: resource requests/limits, probes, security contexts, PodDisruptionBudgets, standard labels, and validation via kubeconform/conftest. Triggers when authoring or reviewing k8s YAML or Helm charts, or before applying manifests to a cluster.
+when_to_use: เขียน/รีวิว Deployment/Service/Helm chart, จะ apply manifest ขึ้น cluster, หรือเช็ค best practice ก่อน merge
+---
+
+## When to Use
+
+Invoke this skill when the task involves any of:
+- Authoring a new Deployment / StatefulSet / DaemonSet / Service / Ingress / Helm chart.
+- Reviewing an existing k8s manifest or Helm chart before merge.
+- Anything that will end in `kubectl apply`, `helm install/upgrade`, `kustomize build`, or an Argo/Flux sync.
+
+Do NOT run live `kubectl apply` as part of review. Render and validate offline first. Treat the cluster as the last step the user triggers, not you.
+
+## Steps
+
+Run these in order. For each gate, report PASS / FAIL / N-A with the exact field that is missing — never a vague "looks fine".
+
+**1. Render + structural lint first (fail fast).**
+- Plain YAML: `kubeconform -strict -summary -schema-location default -schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json' manifests/`. `-strict` rejects unknown fields (catches typos like `limts:`).
+- Helm: render real values, do not lint the templates blind — `helm template <release> ./chart -f values.yaml | kubeconform -strict -summary -`. Run once per environment values file (dev/staging/prod) since prod often diverges.
+- Kustomize: `kustomize build overlays/prod | kubeconform -strict -summary -`.
+- Confirm `apiVersion` is GA where one exists (`apps/v1`, `networking.k8s.io/v1`, `policy/v1` for PDB — `policy/v1beta1` is removed in 1.25+).
+
+**2. Resource requests + limits — every container, including initContainers and sidecars.**
+- Require `resources.requests.cpu`, `requests.memory`, `limits.memory` on each container. Missing requests => `BestEffort`/`Burstable` => first to be OOM-killed/evicted under pressure.
+- Set `limits.memory == requests.memory` (memory is incompressible; bursting then getting OOM-killed is worse than a hard cap).
+- Prefer NO `limits.cpu` (or set it generously). A CPU limit throttles via CFS quota and adds tail latency even when the node is idle — request guarantees the floor.
+- Flag any limit that is >4x its request (noisy-neighbor / scheduling-lie risk).
+
+**3. Probes — liveness, readiness, startup.**
+- `readinessProbe`: required. Without it, traffic hits the pod before it can serve => 502s on rollout.
+- `livenessProbe`: required, but must NOT point at the same deep dependency-checking endpoint as readiness. Liveness = "is this process wedged" (cheap, local). A liveness probe that checks the DB will cascade-restart every pod when the DB blips.
+- `startupProbe`: required for slow-booting apps (JVM, migrations). Set `failureThreshold * periodSeconds` to cover worst-case boot, so liveness doesn't kill a pod mid-startup.
+- Verify `initialDelaySeconds`/`timeoutSeconds`/`failureThreshold` are explicit, not relying on the 1s default timeout (too tight for most HTTP handlers).
+
+**4. securityContext + Pod Security Standards (target `restricted`).**
+Pod-level + container-level:
+- `runAsNonRoot: true` and an explicit `runAsUser` (non-zero).
+- `allowPrivilegeEscalation: false`.
+- `readOnlyRootFilesystem: true` (add an `emptyDir` writable mount for `/tmp` and any cache dirs the app actually needs).
+- `capabilities.drop: ["ALL"]`; add back only what is provably needed (e.g. `NET_BIND_SERVICE`).
+- `seccompProfile.type: RuntimeDefault` (required by PSS restricted).
+- Reject `privileged: true`, `hostNetwork`, `hostPID`, `hostIPC`, and `hostPath` mounts unless explicitly justified in the PR.
+
+**5. Availability — PDB + replicas + spread + rollout.**
+- `replicas >= 2` for anything serving traffic (a single replica = guaranteed downtime on node drain/upgrade).
+- A `PodDisruptionBudget` (`policy/v1`) with `minAvailable` or `maxUnavailable`. Gotcha: `minAvailable: 1` with `replicas: 1` makes the node un-drainable forever — use `maxUnavailable: 1` for replicas of 2-3.
+- `topologySpreadConstraints` across `topology.kubernetes.io/zone` (and `kubernetes.io/hostname`) so all replicas don't land on one node/zone.
+- `strategy.rollingUpdate` with sane `maxUnavailable`/`maxSurge`; a `Recreate` strategy on a serving Deployment means a hard outage window.
+
+**6. Standard labels + namespace.**
+- Apply recommended `app.kubernetes.io/*` labels: `name`, `instance`, `version`, `component`, `part-of`, `managed-by`. Service selectors should target these, not ad-hoc keys.
+- `metadata.namespace` set explicitly (never rely on the kubectl context default — a wrong-namespace apply is a silent prod incident).
+- Service/Deployment selector labels must match the pod template labels exactly, or the Service routes to zero endpoints (it will NOT error — just black-holes traffic).
+
+**7. Policy gate (conftest / OPA).**
+- Run org policy as code: `conftest test --policy policy/ <(helm template ./chart -f values-prod.yaml)`.
+- Common deny rules to verify pass: no `:latest` image tags (pin a digest or semver), images from approved registries only, every workload has the gates from steps 2-6.
+
+**8. Emit a checklist report.** One line per gate: `[PASS|FAIL] <gate> — <resource/container> — <missing field or fix>`. Group FAILs at the top. End with the exact commands to re-validate.
+
+## Common Errors
+
+- **`helm lint` passing ≠ valid manifests.** `helm lint` checks template syntax, not the rendered Kubernetes objects. Always pipe `helm template` output through `kubeconform`.
+- **kubeconform green but CRDs skipped.** Without `-schema-location` for the CRD catalog, custom resources are silently skipped, not validated. Watch the summary for `skipped` counts.
+- **Selector/label drift.** Editing pod template labels without updating the Service/Deployment `selector` leaves the workload running but with zero endpoints. No error is raised — only an empty `kubectl get endpoints`.
+- **`readOnlyRootFilesystem: true` with no writable mount.** App crashes on first write to `/tmp`, logs, or cache. Always pair with an `emptyDir`.
+- **`minAvailable: 1` + `replicas: 1`.** Blocks `kubectl drain` / node autoscaler / cluster upgrades indefinitely. Catches teams by surprise during a maintenance window.
+- **CPU limit causing latency.** Throttling shows as p99 spikes with the node nowhere near saturated. If you see CPU limits on a latency-sensitive service, flag it — remove the limit, keep the request.
+- **`livenessProbe` pointed at a downstream dependency.** Turns a transient DB/cache outage into a restart storm across the whole fleet. Liveness must be local-only.
+- **`policy/v1beta1` PodDisruptionBudget.** Removed in k8s 1.25+. Manifest renders fine offline against an old schema but fails on apply to a modern cluster.
+- **One values file ≠ all environments.** Prod values often disable a probe, change replicas, or add a sidecar. Validate every env's rendered output, not just the default.
+
+## Verify
+
+A manifest is production-ready only when ALL of these are true:
+- `kubeconform -strict` passes for every rendered environment with zero `skipped` resources you care about.
+- `conftest test` (or the org policy gate) returns zero failures.
+- Every container (init + sidecar included) has requests, a memory limit, and the security context from step 4.
+- Every serving workload has readiness + liveness probes, `replicas >= 2`, a `policy/v1` PDB, and topology spread.
+- No `:latest` tags; images pinned to digest or semver.
+- Service selector resolves to a non-empty endpoint set (selector labels == pod template labels).
+
+Report each as a checked line. If you cannot run a validator in this environment, say so explicitly and provide the exact command for the user to run — never claim a manifest is validated when it was only eyeballed.

package/skills/llm-eval-harness/SKILL.md

@@ -0,0 +1,63 @@
+---
+name: llm-eval-harness
+description: Builds an evaluation harness for LLM/agent outputs using golden datasets, code-based scorers, and LLM-as-judge, run as a regression gate when prompts, models, or RAG configs change.
+when_to_use: User wants to measure prompt/model/agent quality, stop regressions when changing a prompt or model, set up llm-as-judge or a golden dataset, or go beyond vibes-based testing. NOT for deterministic unit tests of plain code (use write-tests).
+---
+
+## When to Use
+
+Reach for this when output quality is **non-deterministic** and "looks fine" is not good enough:
+
+- Changing a prompt, model version, temperature, tool definitions, or RAG retrieval config and need proof it didn't regress.
+- Setting up llm-as-judge, a golden dataset, or any CI gate that scores model/agent output.
+- A bug report like "answers got worse after we switched models" — you need a number, not a vibe.
+
+Do **NOT** use for deterministic code (a parser, a pure function, an API contract) — that is a normal unit test; use write-tests. The line: if the same input can produce different-but-valid outputs, it's an eval; if there's one correct output, it's a test.
+
+## Steps
+
+1. **Pick the unit of evaluation first.** Decide what one "case" is: a single prompt→completion, a full agent trajectory (tool calls + final answer), or a RAG turn (query + retrieved context + answer). Everything below is keyed to this unit. Don't mix units in one suite.
+
+2. **Build a lean golden dataset (start at 20–50 cases, not 1000).** Pull real cases from production logs/traces, not invented ones — sample across the actual input distribution. Each case is a row with: `id`, `input` (+ any `variables` like retrieved_context), `expected` (or `reference`), and `tags` (e.g. `edge`, `regression`, `pii`). Deliberately seed known edge cases and every past failure. Store as JSONL or CSV under `evals/data/` so diffs are reviewable in git.
+
+3. **Layer metrics cheapest-first; only escalate to a judge when needed.**
+   - **Code scorers (deterministic, free, run first):** exact match, regex/JSON-schema validation, "must contain / must NOT contain" substrings, valid-tool-was-called, refusal-detector, latency/cost budget, output parses as valid JSON. These catch the majority of regressions with zero flakiness.
+   - **Semantic match:** embedding cosine similarity vs. reference, with a calibrated threshold — use for "is this roughly the right answer" when wording varies.
+   - **LLM-as-judge (last resort, for fuzzy quality):** correctness, faithfulness-to-context, helpfulness, tone. Only build a judge for dimensions code can't check.
+
+4. **Make the judge rigorous, not vibes-in-a-trenchcoat.**
+   - Write a **rubric with discrete levels** (e.g. 1–5 or pass/borderline/fail) where each level has a concrete, observable definition. Force the judge to emit `reasoning` BEFORE `score` (CoT lifts agreement).
+   - **Pin the judge: fixed model id + `temperature=0`** (or near-0) so the gate is reproducible.
+   - **Prefer pairwise or reference-based grading over a lonely 1–10 score** — absolute scores drift; "is A better than reference B?" is far more stable. Mitigate position bias by randomizing A/B order.
+   - For faithfulness/RAG, the judge sees the answer + the retrieved context and rules ONLY on "is every claim supported by context" — not on world knowledge.
+
+5. **Calibrate the judge against humans before trusting it.** Hand-label 20–30 cases. Run the judge on the same cases and compute agreement (Cohen's κ or simple % match). If agreement is low, the rubric is ambiguous — tighten level definitions and re-run. Do not ship a judge you haven't calibrated; an uncalibrated judge is a random number generator with a PhD voice.
+
+6. **Wire variable mapping + per-case pass/fail.** Each case template maps dataset columns → prompt variables (e.g. `{{question}}`, `{{context}}`). Define per-case pass = all code scorers pass AND judge score ≥ threshold. Aggregate to a suite-level pass rate and per-tag breakdown.
+
+7. **Run as a deterministic gate in CI and on every prompt/model/config change.** Trigger on changes to prompt files, model id, or RAG config. The runner: load dataset → run candidate system per case → score → compare against the committed **baseline scores file** (`evals/baseline.json`). **Fail the build if pass-rate drops below threshold OR any case in the `regression` tag flips from pass→fail.** Print a per-case delta table (old score → new score) so the regression is obvious in the PR.
+
+8. **Add random-sample probing to surface NEW failures.** The golden set only tests known cases. On a schedule, sample fresh production inputs, run them, and have the judge flag low-quality outputs for human review. This finds failure modes the static dataset can't.
+
+9. **Close the loop: every confirmed failure becomes a golden case.** When probing or production surfaces a bad output, add it to the dataset tagged `regression` with the corrected `expected`. The harness gets stronger over time and the same bug can never silently return.
+
+## Common Errors
+
+- **Judge with `temperature > 0` → non-reproducible gate.** Two runs of the same diff give different verdicts and the gate becomes noise. Pin model + temp=0, and pin the model *version/snapshot* (a silently-updated judge model is itself a regression source).
+- **Same model judges its own output → inflated, biased scores.** Use a different (often stronger) model as judge, or at minimum acknowledge the bias. Never let the system grade its own homework on quality dimensions.
+- **Absolute 1–10 scoring drifts and clusters at 7–8.** Switch to pairwise (vs. reference) or a discrete rubric with anchored definitions. "Rate 1–10" without anchors is the #1 cause of a useless judge.
+- **Position/verbosity bias in pairwise judging.** Judges favor the first option and longer answers. Randomize order per case (and optionally swap-and-average); penalize verbosity explicitly in the rubric if needed.
+- **Tiny or synthetic dataset → green evals, red production.** 5 cherry-picked cases prove nothing. Pull from real traffic and cover the actual input distribution, including the boring/edge tails.
+- **Reaching for the LLM judge when a code scorer would do.** If the spec is "output must be valid JSON with field X" or "must not leak the system prompt," that's a regex/schema check — deterministic, free, flake-free. Don't pay a judge to check what `assert` can.
+- **No committed baseline → "is this better?" is unanswerable.** Without `baseline.json` under version control you can detect a crash but not a regression. Commit baselines and update them deliberately (in a reviewed PR), never silently.
+- **Judge marks a faithful RAG answer wrong because it "knows better."** A faithfulness judge must rule only on support-by-retrieved-context, not on its own world knowledge. State this explicitly in the judge prompt or you'll get false fails.
+- **Flaky scorers fail the gate for the wrong reason.** Network timeouts / rate limits on judge calls look like quality regressions. Add retries + a clear "harness error" vs. "quality fail" distinction so a 429 doesn't block a ship.
+
+## Verify
+
+- Run the full suite twice on the **same** candidate; the score must be **identical** (proves the gate is deterministic). If it isn't, a scorer/judge has hidden randomness.
+- Confirm the gate actually bites: introduce a deliberately bad prompt and check the suite **fails** with a clear per-case delta pointing at the regressed cases.
+- Confirm a known-good change passes and the baseline-comparison table renders old→new deltas.
+- Spot-check 3–5 judge verdicts by hand against the rubric — the judge's `reasoning` should justify its `score`. If reasoning and score disagree, the rubric/prompt needs tightening.
+- Verify CI is wired: the eval job triggers on prompt/model/RAG-config file changes and blocks merge on failure (not just warns).
+- Confirm every `regression`-tagged case maps to a real past failure, and that the latest surfaced failure has been added back into the golden set.

package/skills/manage-client-server-state/SKILL.md

@@ -0,0 +1,94 @@
+---
+name: manage-client-server-state
+description: Sets up server-state with TanStack Query (caching, mutations, optimistic updates, hydration) and picks the right client-state tool; used when wiring data fetching or untangling state.
+when_to_use: When the user wires data fetching/caching, mentions TanStack Query / React Query, useEffect fetch waterfalls, optimistic UI, SSR hydration, or choosing a state manager (Zustand/Context/Redux).
+---
+
+## When to Use
+
+Reach for this skill when the task involves any of:
+- Wiring data fetching/caching, or the prompt mentions **TanStack Query / React Query** (`useQuery`, `useMutation`, `QueryClient`).
+- A component fetches in `useEffect` and stuffs results into `useState` (manual loading/error flags, refetch-on-prop-change, fetch waterfalls).
+- **Optimistic UI** — update the screen before the server confirms, then roll back on failure.
+- **SSR / RSC hydration** in Next.js (App Router) — prefetch on the server, hand off to the client without a refetch flash.
+- Choosing a **client-state** tool: Context vs Zustand vs Redux, or fixing prop drilling / over-rendering.
+
+Skip / hand off if the task is purely **form validation** (use `build-form-validation`) or **component scaffolding/markup** (component skill). This skill is about *where state lives and how it syncs*, not UI structure.
+
+First principle, state the line out loud before coding: **Server state ≠ client state.** Server state is async, owned by the backend, shared, and can go stale (lists, user profile, search results) → TanStack Query. Client state is synchronous, owned by the UI, local (modal open, active tab, form draft, theme) → `useState` / Zustand / Context. Most "state is a mess" bugs are these two being managed by the same tool.
+
+## Steps
+
+1. **Classify every piece of state first.** Grep the file/feature for `useState`, `useEffect`, `useContext`, `dispatch`. For each, ask: does the value originate from an API/DB? → server state, migrate to TanStack Query. Is it ephemeral UI? → leave as client state. Do NOT migrate UI toggles into Query, and do NOT keep fetched data in `useState`.
+
+2. **Provider + client setup (once per app).** Create one `QueryClient` and wrap the tree in `<QueryClientProvider>`. In Next.js App Router, the client MUST be created inside a `"use client"` component via `useState(() => new QueryClient())` (not a module-level singleton) so each request/user gets its own cache. Set sane defaults: `staleTime` 30–60s for most reads (0 means "always refetch on mount/focus" — usually not what you want), keep `gcTime` (v5; was `cacheTime`) default 5min unless memory-bound.
+
+3. **Replace `useEffect`+`fetch` with `useQuery`.** Pattern:
+   ```ts
+   const { data, isPending, isError, error } = useQuery({
+     queryKey: ['todos', { status, page }], // serializable, hierarchical
+     queryFn: ({ signal }) => fetchTodos({ status, page, signal }),
+     staleTime: 30_000,
+   })
+   ```
+   Delete the manual `loading`/`error`/`data` `useState` and the `useEffect`. Pass `signal` into fetch for auto-cancel. **queryKey strategy:** array form, broad→narrow (`['todos']` → `['todos', id]` → `['todos', { filters }]`). Every variable the `queryFn` reads MUST appear in the key — missing deps = stale data served for the wrong params. Centralize keys in a `queryKeys` factory object to avoid typos and make invalidation greppable.
+
+4. **Mutations + invalidation.** Use `useMutation` for writes. After success, invalidate the affected reads so they refetch:
+   ```ts
+   const qc = useQueryClient()
+   useMutation({
+     mutationFn: updateTodo,
+     onSuccess: () => qc.invalidateQueries({ queryKey: ['todos'] }),
+   })
+   ```
+   `invalidateQueries({ queryKey: ['todos'] })` matches all keys *prefixed* by `['todos']` — that's the lever the hierarchical key design buys you. Prefer invalidation over manually hand-editing the cache unless you're doing optimistic updates.
+
+5. **Optimistic updates (only where latency is felt — toggles, likes, reorder).** The four-callback contract:
+   ```ts
+   useMutation({
+     mutationFn: toggleTodo,
+     onMutate: async (next) => {
+       await qc.cancelQueries({ queryKey: ['todos'] })      // stop in-flight refetch clobbering us
+       const prev = qc.getQueryData(['todos'])               // snapshot for rollback
+       qc.setQueryData(['todos'], (old) => applyOptimistic(old, next))
+       return { prev }                                       // context passed to onError
+     },
+     onError: (_e, _next, ctx) => qc.setQueryData(['todos'], ctx?.prev), // rollback
+     onSettled: () => qc.invalidateQueries({ queryKey: ['todos'] }),     // reconcile w/ server
+   })
+   ```
+   All three of `cancelQueries` / snapshot+rollback / `onSettled` invalidate are required — drop any one and you get flicker, lost rollback, or permanent drift from the server.
+
+6. **SSR / RSC hydration (Next.js App Router).** In the **server component**: create a request-scoped `QueryClient`, `await queryClient.prefetchQuery({ queryKey, queryFn })`, then render `<HydrationBoundary state={dehydrate(queryClient)}>` wrapping the client component. The client component calls `useQuery` with the **identical queryKey** — it reads the dehydrated cache, no refetch, no loading flash. Mismatched keys between prefetch and `useQuery` = silent double-fetch on the client; this is the #1 hydration bug, verify keys are byte-identical.
+
+7. **Pick the client-state tool (the non-server state from step 1):**
+   | Need | Use |
+   |---|---|
+   | Low-frequency, rarely-changing (theme, locale, auth user object) | **Context** |
+   | Cross-cutting, frequently-updated, shared across distant components (cart, wizard, filters) | **Zustand** |
+   | Local to one subtree | keep `useState` / `useReducer`, lift only as far as needed |
+   | Complex shared logic + time-travel/devtools/middleware genuinely needed | **Redux Toolkit** (default to NOT this) |
+   Do **not** reach for Redux by reflex, and do **not** put server data in any of these — that's step 3's job.
+
+8. **Kill prop drilling + unnecessary re-renders.** If a prop is threaded through 3+ layers only to reach a leaf, move it to Context or Zustand. For Zustand, **always select narrowly** — `useStore((s) => s.cart.count)`, never `useStore((s) => s.cart)` or the whole store, or every store change re-renders the component. For Context, split into multiple providers (e.g. separate `ThemeContext` and `AuthContext`) so a change to one doesn't re-render consumers of the other; memoize the provider `value`.
+
+## Common Errors
+
+- **`queryKey` missing a `queryFn` dependency** → wrong/stale data shown when params change. Every variable used in `queryFn` must be in the key.
+- **Module-level `new QueryClient()` in Next.js** → cache leaks across requests/users on the server (one user sees another's data). Create it inside `useState`/per-request.
+- **`v5` rename trap:** `cacheTime` → `gcTime`, `isLoading` → `isPending` (for "no data yet"), `useQuery` no longer takes positional args (options object only), callbacks `onSuccess/onError` removed from `useQuery` (still on `useMutation`). If you see those on a query, it's v4 code.
+- **`staleTime: 0` (default) + refetchOnWindowFocus** → app hammers the API on every tab switch. Set a real `staleTime` for reads that don't need to be live.
+- **Optimistic update without `cancelQueries`** → an in-flight background refetch resolves *after* your optimistic `setQueryData` and overwrites it → UI flickers back. Always `cancelQueries` in `onMutate`.
+- **Hydration key mismatch** → server prefetches `['todos']`, client `useQuery(['todos', filters])` → no cache hit, refetch + flash. Keys must match exactly.
+- **Subscribing to the whole Zustand store** → re-renders on unrelated state changes. Use a selector.
+- **`invalidateQueries` with an over-narrow key** → sibling queries stay stale. Invalidate at the right prefix level.
+- **Putting server data in Zustand/Context "to share it"** → you reimplement caching/invalidation/refetch badly. Let TanStack Query own server state; share the *query*, not a copy.
+
+## Verify
+
+- Search the touched files: no `fetch(`/axios call inside `useEffect` writing to `useState` remains for server data; those are now `useQuery`/`useMutation`.
+- TypeScript/build passes; no v4-only API names (`cacheTime`, `useQuery({ onSuccess })`) left.
+- Open React Query Devtools: each screen's queries appear with sensible keys, correct `fresh`/`stale` status, and no duplicate keys fetching the same data.
+- Mutate something: the relevant query auto-refetches (invalidation works). For optimistic paths, throttle the network and confirm UI updates instantly then either persists or **rolls back** on a forced 500.
+- SSR pages: disable JS or check the Network tab — initial data is in the server HTML and the client does **not** refetch on mount (no loading flash, no duplicate request).
+- Profile a noisy interaction (React DevTools Profiler): components not consuming the changed slice do **not** re-render after the Context/Zustand split + selectors.

package/skills/mermaid-diagram/SKILL.md

@@ -0,0 +1,61 @@
+---
+name: mermaid-diagram
+description: Turn requirements, code, or a system description into validated Mermaid diagrams (flowchart, sequence, class, ER, state, C4, Gantt, mindmap, git graph) and verify they render via mermaid-cli before delivering.
+when_to_use: User asks to diagram/visualize a flow, architecture, data model, state machine, or sequence; needs a chart embedded in docs/markdown; "draw me the flow of X". Skip when the user wants an image edited or a non-Mermaid format (e.g. raw SVG, PlantUML, Graphviz/DOT) — those are different tools.
+---
+
+## When to Use
+
+Trigger when the intent is "show this as a diagram" and the target is Mermaid (renders natively in GitHub, GitLab, Obsidian, VS Code, most docs). Map intent → diagram type up front; picking wrong wastes a render cycle:
+
+| Intent / phrasing | Diagram type | Header keyword |
+|---|---|---|
+| Steps, branches, "the flow of X", decision logic | flowchart | `flowchart TD` / `LR` |
+| Actors talking over time, API calls, request/response | sequence | `sequenceDiagram` |
+| DB schema, tables + relations + cardinality | ER | `erDiagram` |
+| OOP model, types, fields, inheritance | class | `classDiagram` |
+| Lifecycle, status machine, "states of an order" | state | `stateDiagram-v2` |
+| System / container / service boundaries | C4 | `C4Context` / `C4Container` |
+| Timeline, project phases, dependencies-over-time | gantt | `gantt` |
+| Hierarchy of ideas, brainstorm tree | mindmap | `mindmap` |
+| Branch/merge history | git graph | `gitGraph` |
+
+If unsure between flowchart and sequence: **does order-of-time + who-does-what matter?** yes → sequence, no → flowchart.
+
+## Steps
+
+1. **Read the source of truth, don't guess.** If diagramming code, open the actual files/functions and trace real control flow, call edges, or schema — not an assumed design. For ER/class, pull field names and FKs from the real schema/models.
+2. **Pick the type** from the table above and the direction. Default `flowchart TD` (top-down) for processes; `LR` (left-right) when there are many sequential steps (wide reads better than tall).
+3. **Write the Mermaid source to a temp file**, e.g. `/tmp/diagram.mmd` — do not hand-build it inline; you need a file to validate.
+4. **Validate by rendering** — this is non-negotiable, a diagram that doesn't parse is a failure:
+   ```bash
+   npx -y @mermaid-js/mermaid-cli@latest -i /tmp/diagram.mmd -o /tmp/diagram.svg
+   ```
+   - First run downloads the package; that's expected. If `mmdc` is already on PATH, use it directly: `mmdc -i in.mmd -o out.svg`.
+   - On headless/CI or if Chromium fails to launch, add a puppeteer config: write `{"args":["--no-sandbox"]}` to `/tmp/pp.json` and append `-p /tmp/pp.json`.
+   - SVG is the cheapest/fastest output for a syntax check. Use `-o out.png` only if the user wants a raster image.
+5. **If it fails, fix the root cause and re-render.** Read the parser error (it gives a line/token), correct the syntax, run step 4 again. Loop until it renders clean. Never "soften" by deleting the failing node or wrapping problem text — fix the actual syntax (see Common Errors).
+6. **Tighten for readability** before delivery: short node labels (offload detail to surrounding prose), `subgraph` to group related nodes, consistent edge labels, and a sane direction. A correct-but-unreadable diagram still fails the task.
+7. **Deliver as a fenced ```mermaid block** ready to paste into markdown/Obsidian. Do not deliver the SVG path as the answer unless the user explicitly asked for an image file — the fenced source is the product.
+
+## Common Errors
+
+- **Parens/brackets/quotes inside a label break the parser.** `A[Fetch (cached)]` fails. Wrap the whole label in double quotes: `A["Fetch (cached)"]`. Same fix for `:`, `;`, `#`, `&`, `<`, `>`, `/`.
+- **HTML in labels needs `<br/>` not `\n`.** Newlines inside a node use `A["Line one<br/>Line two"]`. A literal `\n` renders as text.
+- **Reserved word `end`** as a lowercase node id silently corrupts flowcharts. Use `End`, `END`, or `e_end`.
+- **Sequence diagrams: declare or imply participants consistently.** Mixing `participant A as Auth` then referring to `Auth` later (the alias, not the id) breaks. Reference the **id** in arrows, the alias is display-only.
+- **ER cardinality syntax is strict:** `USER ||--o{ ORDER : places` (one-to-many). Common crash is using `1` / `*` instead of the `||`, `o{`, `}o`, `|{` tokens.
+- **C4 diagrams need the right header** (`C4Context`, `C4Container`, `C4Component`) and `Rel(a, b, "label")` — they do NOT use flowchart arrow syntax. Don't mix the two grammars.
+- **`gantt` dates must be ISO** (`YYYY-MM-DD`) and need a `dateFormat` line; freeform dates fail.
+- **`%%` is the comment marker, not `//` or `#`.** A stray `//` comment is parsed as a node and throws.
+- **`graph` is the legacy keyword; prefer `flowchart`.** Both parse, but `flowchart` gets newer features (e.g. `&` chaining). Don't mix dialects in one block.
+- **Indentation matters in `mindmap`** — it's whitespace-structured like YAML, not bracket-structured. Inconsistent indent = wrong tree.
+
+## Verify
+
+Done only when ALL hold:
+- [ ] `mmdc`/`npx ... mermaid-cli` exited 0 on the final source (an actual render happened — not "it looks right").
+- [ ] Diagram type matches the user's intent (flow vs sequence vs schema), not just "a diagram."
+- [ ] Every node/edge in the diagram maps to something real in the source code/spec — no invented steps, no dropped branches.
+- [ ] Labels are concise and the layout direction reads cleanly; groups use `subgraph` where it reduces crossing edges.
+- [ ] Delivered as a copy-paste-ready ```mermaid fenced block (plus the image path only if an image was requested).