@coralai/sps-cli 0.41.2 → 0.43.0

package/skills/devops/references/iac.md

@@ -0,0 +1,252 @@
+# Infrastructure-as-Code
+
+Terraform, Pulumi, CDK, CloudFormation. Patterns, not syntax.
+
+## Principles
+
+1. **Code, not console.** Every production resource is declared in code. Console use is for exploration only, not for shipping.
+2. **State is sacred.** Losing or corrupting state means reconstructing reality from the cloud. Store it remotely, lock it.
+3. **Plan before apply.** `terraform plan` (or equivalent) is a read-only dry run. Diff it; understand it; only then apply.
+4. **Modules for reuse, not for abstraction.** A module that no one else uses is just folders. Extract when you have two callers, not before.
+5. **Environments as instances of a config.** Same code, different variables. Dev, staging, prod shouldn't fork into three different trees.
+6. **Less is more.** The smallest resource set that meets requirements. Every extra resource is an extra blast radius.
+
+## Tool choice
+
+| Tool | Strengths |
+|---|---|
+| **Terraform / OpenTofu** | Multi-cloud, huge provider ecosystem, mature |
+| **Pulumi** | Real programming language; complex logic feels natural |
+| **AWS CDK** | AWS-native, TypeScript / Python, feels like SDK |
+| **CloudFormation** | AWS-native, declarative, tight integration |
+| **Ansible** | Imperative config mgmt (VMs), not declarative infra |
+| **Kubernetes manifests / Helm / Kustomize** | K8s-specific |
+| **Crossplane / KRO** | K8s-native for cloud infra |
+
+Pick one IaC tool per cloud estate. Using three for one codebase creates state duplication and drift.
+
+## State — remote, locked
+
+Local state on a laptop is a failure mode waiting to happen. Remote backend + locking is non-negotiable for team work.
+
+Terraform remote backends:
+- **S3 + DynamoDB** (AWS) — classic, cheap.
+- **Terraform Cloud / HCP Terraform** — managed, VCS integration.
+- **GCS + lock** (GCP).
+- **Azure Storage** (Azure).
+
+Enable:
+- Encryption at rest.
+- Versioning (so a corrupt state can be rolled back).
+- Restricted access (only CI + admins).
+
+## One state file vs. many
+
+Split state when:
+- **Blast radius**: a typo in one part shouldn't risk another (separate "network" from "apps").
+- **Apply time**: a 40-minute plan is painful; split so changes in one area only plan that area.
+- **Permissions**: different teams own different pieces.
+
+Typical split:
+```
+networking/     — VPCs, subnets, DNS
+data/           — databases, caches, queues
+platform/       — K8s clusters, service mesh
+apps/service-a/
+apps/service-b/
+```
+
+Cross-state references via remote state data sources:
+
+```hcl
+data "terraform_remote_state" "network" {
+  backend = "s3"
+  config  = { bucket = "...", key = "networking.tfstate", region = "..." }
+}
+
+resource "aws_instance" "app" {
+  subnet_id = data.terraform_remote_state.network.outputs.subnet_id
+}
+```
+
+## Modules
+
+Reuse is an effect; don't modularize for the sake of it.
+
+```
+modules/
+├── s3-bucket/       # simple, focused
+├── rds-postgres/    # multiple call sites; worth the abstraction
+└── vpc/             # canonical
+```
+
+Rules:
+- Module inputs: the minimum variables that matter; sensible defaults for the rest.
+- Module outputs: only what callers need.
+- Version modules (git tag / registry version) if shared across repos.
+
+Anti-pattern: "god module" with 50 inputs covering every hypothetical case. That's configuration disguised as code.
+
+## Workspaces vs. env-specific folders
+
+### Workspaces (Terraform)
+
+Same code, different state per workspace.
+
+```
+terraform workspace new dev
+terraform workspace new prod
+```
+
+OK for dev/prod parity when the topology is truly identical.
+
+### Folder-per-env (often preferred)
+
+```
+environments/
+├── dev/
+│   └── main.tfvars
+├── staging/
+│   └── main.tfvars
+└── prod/
+    └── main.tfvars
+```
+
+Each env has its own state and vars file. Same underlying modules. Explicit, reviewable, allows env-specific overrides.
+
+## Review flow
+
+Every IaC change is a PR. `terraform plan` output in the PR description or CI comment. Reviewer reads the plan and the code.
+
+Automate it:
+
+```yaml
+- run: terraform init
+- run: terraform plan -out=plan.tfplan
+- run: terraform show -no-color plan.tfplan > plan.txt
+- uses: actions/github-script@v7
+  with: { script: |
+    const plan = fs.readFileSync('plan.txt', 'utf8');
+    await github.rest.issues.createComment({ ..., body: '```\n' + plan + '\n```' });
+  }
+```
+
+Prod apply gated behind approval:
+
+```yaml
+environment:
+  name: prod
+  url: https://...
+```
+
+## Drift
+
+State says one thing; reality says another (someone clicked in the console, an external process modified a resource).
+
+- `terraform plan` detects drift.
+- Reconcile: revert console changes or adopt them into code.
+- Policy: disable console write access for prod; force all changes through IaC.
+
+Drift ignored becomes a permanent parallel state that diverges further every day.
+
+## Secret handling in IaC
+
+- Secrets don't live in `.tfvars` committed to git.
+- Pull from secret manager at apply time (`data "aws_secretsmanager_secret_version"`).
+- Output sensitive values with `sensitive = true` so they don't leak in logs.
+- `.tfstate` itself contains sensitive values — protect the backend.
+
+## Tagging strategy
+
+Every cloud resource gets tags. Makes cost allocation, search, ownership unambiguous.
+
+```hcl
+default_tags = {
+  environment = var.env
+  service     = var.service
+  owner_team  = var.team
+  managed_by  = "terraform"
+  repo        = "github.com/.../infra"
+}
+```
+
+Enforce via policy (SCPs on AWS, Azure Policy, custom `terraform-compliance`).
+
+## Avoid `count` / `for_each` on resources likely to change order
+
+Terraform tracks resources by address. `aws_instance.web[0]` is not the same as `aws_instance.web[1]`. If you insert into the middle, everything after is "new".
+
+Prefer `for_each` over `count` — keyed by a string, stable under insertion.
+
+```hcl
+# ❌ count — fragile if order changes
+resource "aws_instance" "web" {
+  count = length(var.regions)
+  region = var.regions[count.index]
+}
+
+# ✅ for_each — keyed
+resource "aws_instance" "web" {
+  for_each = toset(var.regions)
+  region   = each.value
+}
+```
+
+## Lifecycle rules
+
+```hcl
+lifecycle {
+  prevent_destroy = true     # guard prod databases, S3 buckets with data
+  create_before_destroy = true  # zero-downtime replacement where possible
+  ignore_changes = [tags["last_updated"]]  # don't churn on external tag writes
+}
+```
+
+`prevent_destroy` on anything that would be irrecoverable to delete. Deliberate override needed to actually destroy.
+
+## Blast radius
+
+Running `terraform destroy` in the wrong directory has wiped real infrastructure. Mitigations:
+- Separate folders per env with distinct state backends.
+- Prod destroy requires a separate pipeline or a specific role.
+- Critical resources have `prevent_destroy`.
+
+## Policy as code
+
+Test that the plan adheres to rules before apply.
+
+- **OPA / Conftest** — general-purpose policy, plaintext rules.
+- **Terraform Sentinel** (HCP Terraform) — policy as code.
+- **Checkov, tfsec, terrascan** — pre-built security rules (public S3 buckets, encryption, etc.).
+
+Typical guards:
+- No public S3 buckets.
+- No unencrypted RDS / EBS.
+- No 0.0.0.0/0 ingress on non-frontend services.
+- All resources have required tags.
+
+## Rollback
+
+IaC rollback = revert the commit + apply. It works when the infra change is self-contained.
+
+It does NOT work for:
+- Data migrations (schema changes, in-place transformations).
+- Resources that were deleted with associated data.
+- Stateful upgrades where the backing engine doesn't support downgrade.
+
+For those, plan forward-only fixes.
+
+## Anti-patterns
+
+| Anti-pattern | Fix |
+|---|---|
+| Local state | Remote + locking |
+| No module versioning for shared modules | Tag + version pin |
+| Env-specific logic via `if env == "prod"` | Env-specific `.tfvars` / folder |
+| Every change runs `apply` without review | PR + plan output |
+| Sensitive outputs without `sensitive = true` | Mark them |
+| Giant 5000-line main.tf | Split by resource group / module |
+| `terraform taint` as a regular workflow | Fix the root cause; taint is a hack |
+| `null_resource` + `local-exec` for everything | Find a proper provider |
+| Hand-written policies enforced by discipline | Automate with OPA / tfsec |
+| Cloud console changes that "just need to happen" | Update IaC; revert the console |

package/skills/devops/references/observability.md

@@ -0,0 +1,228 @@
+# Observability (Platform)
+
+Log / metric / trace pipelines, alerting, on-call, runbooks. For app-level signal definition, see `backend/references/observability.md`; this file covers the platform plumbing.
+
+## The stack
+
+```
+App  ──stdout──▶  Collector (fluent-bit, otel-collector, vector)
+     ──metric──▶  Prometheus / Cloud Monitoring / Datadog / NewRelic
+     ──trace───▶  OpenTelemetry Collector ──▶ Jaeger / Tempo / DD APM
+
+Alerting: Prometheus Alertmanager / Grafana / PagerDuty / OpsGenie
+```
+
+Pick the right number of tools. Four different tools with overlapping coverage is a tax; one plus another specialist is usually enough.
+
+## Logs
+
+### Collection
+
+- Container logs → stdout/stderr.
+- Daemon on each node reads container logs (`fluent-bit`, `fluentd`, `vector`, cloud-native).
+- Collector forwards to the backend (Elasticsearch, Loki, Datadog Logs, Cloud Logging).
+
+Don't write logs to local files inside containers. Lost on pod restart; hard to collect.
+
+### Format
+
+JSON. Every line a structured event. See `backend/references/observability.md` for field names.
+
+### Retention
+
+Tier by age:
+- **Hot**: 7–14 days, fast search.
+- **Warm**: 30–90 days, slower but still searchable.
+- **Archive**: 1+ year, S3 / cold storage, restore on demand.
+
+Log volume grows with traffic; set retention per env (dev can be 3 days, prod 30). Otherwise, the bill does the planning for you.
+
+### Sensitive data
+
+Redact at source — the app's logger, not the collector. Once a secret hits the pipeline it's harder to control.
+
+Check your logs periodically for leaked PII / tokens. Automated scanning rules (pattern matching JWT, credit card) in the pipeline.
+
+## Metrics
+
+### Collection
+
+- **Pull** (Prometheus) — scraper hits app endpoints.
+- **Push** (StatsD, OTLP) — app pushes to a gateway / collector.
+
+Pull scales well at moderate cluster sizes, gets fiddly at huge scale. Push is simpler at scale but loses some visibility.
+
+### Standards
+
+OpenTelemetry (OTel) is becoming the de-facto standard for metric + trace instrumentation. Instrument once with OTel SDKs; switch backends by changing the collector config.
+
+### Cardinality
+
+Every unique combination of label values creates a new time series. High-cardinality labels (user_id, request_id) blow up storage and cost.
+
+```
+# ✅ bounded
+http_requests_total{service="api", route="/orders", method="POST", status="200"}
+
+# ❌ unbounded
+http_requests_total{service="api", user_id="u_01HX..."}
+```
+
+The cloud will silently charge you for cardinality. Watch the count of series.
+
+### Four golden signals (per service)
+
+1. **Latency** — how long do requests take (p50/p95/p99)?
+2. **Traffic** — how many requests per second?
+3. **Errors** — rate of failed requests?
+4. **Saturation** — how full is it? (CPU, queue depth, connection pool)
+
+Dashboards start here. Drill into specifics from the starting point.
+
+## Traces
+
+OpenTelemetry instrumented endpoints + propagated context.
+
+```
+Request ─▶ Service A [span] ─▶ Service B [span] ─▶ DB [span]
+```
+
+Each span has timing, tags, events. Together they form the request timeline.
+
+### Sampling
+
+Head-based (per request, decide at ingress):
+- 1–10% typical.
+- Boost to 100% for errors.
+
+Tail-based (sample after seeing the whole trace):
+- Keep slow traces, error traces, unusual patterns.
+- Needs a full collector layer (otel-collector).
+
+Tracing overhead is real — don't trace 100% in prod without tail-based sampling.
+
+## Dashboards
+
+### Structure
+
+One dashboard per service, standard layout:
+- Overview: RED metrics (Rate, Errors, Duration).
+- Saturation: CPU, memory, pool utilization.
+- Dependencies: DB, cache, upstream services.
+- Recent deploys marked as vertical annotations.
+
+Links to runbook + logs + traces.
+
+### Don't build 50 dashboards
+
+Most go stale within weeks. Focus on a small set that matters:
+- One per critical service.
+- One per SLO.
+- A few investigative templates ("compare p99 before/after a given deploy").
+
+## Alerts
+
+### Principles
+
+- **Alert on symptoms, not causes.** "Users can't check out" beats "CPU is 80%".
+- **Every alert is actionable** — there's a specific thing the oncall does.
+- **Every alert has a runbook** linked in the alert body.
+- **Every alert has an owner** — the team / service that owns the fix.
+
+### Severity levels
+
+- **P1 / SEV-1**: page the oncall; revenue / customer-facing impact.
+- **P2 / SEV-2**: notify in team channel; degraded state.
+- **P3 / SEV-3**: track as an issue; investigate next business day.
+
+Only P1 should wake someone up. Too many P1s → pager fatigue → missed alerts.
+
+### Tuning
+
+- Alert fires → wasn't actionable → either tune the threshold or delete it.
+- Alert fires at 3am and auto-resolves at 3:15am with no action → wasn't actionable.
+- Alert with "click dashboard, maybe it's fine" → wasn't actionable.
+
+Audit monthly.
+
+## On-call
+
+### Rotation
+
+- Weekly rotation typical; one primary + one secondary.
+- Handoff meeting: what's ongoing, what's worrying.
+- On-call participants must have access: can deploy, rollback, scale.
+
+### Triage flow
+
+1. **Acknowledge** — clock is ticking on MTTR.
+2. **Stop the bleeding** — rollback, scale up, disable a feature flag. Don't perfect-fix in the moment.
+3. **Gather context** — what changed recently? dashboards, logs, traces.
+4. **Escalate** — bring in the service owner if you're not them.
+5. **Post-incident** — see below.
+
+### Post-incident
+
+Every P1 gets a postmortem. Blameless.
+
+Template:
+- **Summary** — one paragraph.
+- **Impact** — who / how much / how long.
+- **Timeline** — minute-by-minute of detection, response, resolution.
+- **Root cause** — technical + process.
+- **Action items** — specific, owned, dated.
+- **Lessons** — what was surprising.
+
+Track action items to completion. Unshipped postmortem actions are how the same incident happens twice.
+
+## SLO / error budget
+
+Set SLOs (service-level objectives) that match user expectations. Derive error budget.
+
+```
+SLO: 99.9% of /orders POST succeed in ≤ 500 ms
+Budget: 0.1% × 30 days ≈ 43 min / month
+```
+
+Burn rate:
+- Slow burn — spend budget over weeks (minor quality erosion).
+- Fast burn — exhaust weekly budget in a day (real problem).
+
+Alert on burn rate, not just on individual failures. "We're burning budget 10× too fast" is actionable.
+
+## Health endpoints
+
+```
+/health/live   — process alive
+/health/ready  — can serve traffic (DB reachable, cache reachable)
+```
+
+Container orchestrators use both:
+- Live failing → restart the container.
+- Ready failing → take out of load balancer, leave alive.
+
+Never put business logic in health checks. Keep them cheap and boring.
+
+## Cost visibility
+
+Observability is expensive at scale. Monitor the bill:
+- Logs ingested per service per day.
+- Metric series count.
+- Trace spans per second.
+
+When one service exports 10× what others do — investigate. Usually debug logging left on, or a metric with a user-id label.
+
+## Anti-patterns
+
+| Anti-pattern | Fix |
+|---|---|
+| Logs written to files in containers | stdout |
+| Alerts on infrastructure without symptom mapping | Alert on service impact |
+| Dashboards nobody reads | Delete unused; focus on core ones |
+| Runbook-less alerts | Every alert links a runbook |
+| Tracing 100% in prod without sampling | Head or tail sampling |
+| Metric labels on request IDs | Use logs/traces for high-cardinality |
+| "I'll set up monitoring later" | Observability before launch |
+| Alert channel drowning in noise | Audit and tune |
+| No oncall → whoever's free panics | Formal rotation, documented escalation |
+| Postmortems as blame sessions | Blameless format; focus on systems and actions |

package/skills/devops/references/secrets.md

@@ -0,0 +1,178 @@
+# Secrets
+
+Storage, rotation, access, scanning. The part that goes wrong quietly.
+
+## The rules
+
+1. **Never in source control.** Ever. `.env` files stay in `.gitignore`; secrets come from a manager.
+2. **One secret, one owner.** Scoped per service, per env. The "shared creds" bucket is a leak waiting.
+3. **Rotate on schedule AND on compromise.** Short lifetime = small window of exposure.
+4. **Least privilege.** A service key can read its own DB, not every DB.
+5. **Audit every read.** You should be able to tell who accessed prod secret X yesterday.
+6. **Encrypt at rest AND in transit.** Default in modern secret managers; verify.
+
+## Where to store them
+
+| Tool | For |
+|---|---|
+| **AWS Secrets Manager / Parameter Store** | AWS; IAM-integrated |
+| **GCP Secret Manager** | GCP; IAM-integrated |
+| **Azure Key Vault** | Azure; RBAC-integrated |
+| **HashiCorp Vault** | Cloud-agnostic; dynamic secrets, rich ACLs |
+| **1Password / Bitwarden** | Human-held secrets, small teams |
+| **Kubernetes Secrets** | K8s-native, lightweight; encrypt etcd |
+| **Sealed Secrets / SOPS** | Encrypt secrets that live in git (controller decrypts) |
+
+Pick one canonical store per cloud estate. Sprawl breeds drift — the same secret in three places rotates in one.
+
+## Access patterns
+
+### At deploy
+
+- CI has scoped permission to fetch the secrets its job needs (e.g., `DEPLOY_ROLE_PROD`).
+- Terraform pulls via `data` sources at apply (not baked into `.tfvars`).
+- Kubernetes: use the cloud provider's secret CSI driver or External Secrets Operator.
+
+### At runtime
+
+- Container pulls secrets from the manager on startup via the platform (IRSA on EKS, Workload Identity on GKE, IAM on Azure).
+- Or mount as files; the app reads from `/secrets/db_url`.
+- Never bake into the image.
+
+### Locally
+
+- Developers authenticate to the secret manager (SSO + session).
+- CLI pulls secrets on demand: `op run --env-file .env.tpl -- npm start`.
+- Don't ship shared `.env.dev` files on Slack.
+
+## Rotation
+
+For every secret, know:
+- Who rotates (automated job, human?).
+- How often (14 d? 90 d?).
+- How do consumers pick up the new value without downtime?
+
+Rotation patterns:
+
+### Dual-secret window
+
+1. Create secret v2 alongside v1.
+2. Consumers accept both.
+3. Update producers to use v2.
+4. Disable v1 after grace period.
+
+Zero downtime if all consumers support reading both.
+
+### Break-glass
+
+Some secrets (signing keys, root creds) rarely rotate. Document:
+- Access procedure (break-glass requires 2-person approval).
+- Rotation playbook.
+- Who to notify.
+
+## Dynamic secrets
+
+The gold standard. Vault generates short-lived DB credentials on request; they expire in minutes / hours.
+
+```
+app → Vault: "give me DB creds for service X"
+Vault → DB: CREATE USER temp_xyz WITH GRANT ...
+Vault → app: { user: "temp_xyz", pass: "...", ttl: 1h }
+# after 1h Vault revokes the user
+```
+
+A leaked cred is useless after an hour. Requires upfront setup; worth it for sensitive DBs.
+
+## Pre-commit scanning
+
+Catch secrets before they hit the repo.
+
+- **gitleaks** / **detect-secrets** in pre-commit hook.
+- **trufflehog** / **gitleaks** in CI, scanning history.
+- If something lands by accident: rotate immediately, don't just `git revert`. History is forever.
+
+```yaml
+# pre-commit
+- repo: https://github.com/gitleaks/gitleaks
+  rev: v8.18.0
+  hooks: [{ id: gitleaks }]
+```
+
+## Git history cleanup — last resort
+
+If a secret is in the history:
+1. **Rotate the secret immediately.** Treat as compromised.
+2. Cleaning history (`git filter-repo`, BFG) is partial — anyone with the old clone still has it.
+3. Force-push is disruptive; coordinate with team.
+4. Document the incident.
+
+Assume: if it was pushed, someone scraped it already.
+
+## Encrypted configs in git (SOPS)
+
+For teams that want encrypted secrets checked into the repo (mostly for smaller teams / K8s manifests):
+
+```
+config.yaml:         # SOPS-encrypted at rest
+  db:
+    url: ENC[AES256_GCM,data:abc123...]
+```
+
+SOPS uses a KMS key (AWS KMS, GCP KMS, age) to decrypt at runtime. Team members with access to the KMS key can decrypt.
+
+Rules:
+- Encrypt secret fields only (`sops.encrypted_regex: '^(password|token|url)$'`).
+- Commit `.sops.yaml` describing the key.
+- Revoke KMS key access when a team member leaves.
+
+## Secret sprawl — audit
+
+Once a quarter, audit:
+- How many secrets exist?
+- Who / what has access to each?
+- When was each last rotated?
+- Any that are unused (zero accesses in 90 d)?
+
+Cleanup: revoke unused keys. Deleted secrets can't leak.
+
+## Non-secret configs
+
+Not everything is a secret. Feature flags, service endpoints, log levels are config — check them into code (env-specific files), don't put them in the secret manager.
+
+Mixing bloats the secret manager and trains people to ignore the "secret" marker.
+
+## Logging and secrets
+
+- Redact at the logger — don't rely on calls to `log.info(token[:5] + '...')`.
+- Structured loggers (Winston, Pino, Zap, slog) support redaction on field names.
+- Test: grep logs for known secret prefixes. Zero hits.
+
+## Cloud IAM vs. shared secrets
+
+Prefer IAM / workload identity over shared long-lived API keys.
+
+```
+# ❌
+AWS_ACCESS_KEY_ID=AKIA...
+AWS_SECRET_ACCESS_KEY=...
+
+# ✅
+# Pod assumes a role via IRSA/Workload Identity; temporary creds minted per request.
+```
+
+Same for DB access (RDS IAM auth), MQ (IAM policies), object storage. If the cloud supports workload identity, use it.
+
+## Anti-patterns
+
+| Anti-pattern | Fix |
+|---|---|
+| Secrets in `.env` committed to git | `.gitignore` + secret manager |
+| Same API key used by every service | Scope per service |
+| Rotating by "reminding the team once a year" | Automate or track with a rotation job |
+| Logging tokens for debugging | Redact; use opaque IDs |
+| Long-lived cloud API keys in CI | OIDC + short-lived role assumption |
+| Shared "admin" DB user per team | Per-user creds (even for humans), audit log |
+| Decrypting secrets to an env var only to forget | Let the app read from a file or secret mount |
+| Secret "just this one time" pasted in a ticket | Rotate now; use a secure channel (short-lived link) |
+| No alerts on secret access from unusual IPs / times | Enable Cloud audit logs + alert |
+| Skipping pre-commit scanning "it slows me down" | It saves an incident |