@synapta/skills 0.1.0 → 0.1.2

package/skills/deploy-k8s/references/observability.md

@@ -0,0 +1,302 @@
+# Observability
+
+**Directive:** When generating or reviewing any production workload, ALWAYS include metrics exposure, structured logging, and health probes. Observability is not optional -- if you cannot measure it, you cannot operate it. Default security posture is PSS "restricted" profile.
+
+## When to use
+
+Consult this reference whenever the task involves:
+- Deploying any workload to a production or staging cluster
+- Setting up monitoring, alerting, or dashboards
+- Investigating incidents or performing post-mortems
+- Capacity planning or performance analysis
+- Configuring log aggregation or distributed tracing
+
+---
+
+## Probes as the Foundation
+
+Liveness, readiness, and startup probes are the most basic form of observability -- they tell Kubernetes whether your application is alive, ready, and initialized. See **fragile-rollouts.md** for detailed probe configuration rules. Without correct probes, no amount of metrics or logging will prevent cascading failures.
+
+---
+
+## Prometheus Metrics Exposure
+
+### Annotations pattern (works without prometheus-operator)
+
+Add annotations to the Pod template so Prometheus discovers and scrapes the target:
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: order-service
+  namespace: orders
+spec:
+  replicas: 3
+  selector:
+    matchLabels:
+      app: order-service
+  template:
+    metadata:
+      labels:
+        app: order-service
+      annotations:
+        prometheus.io/scrape: "true"
+        prometheus.io/port: "9090"
+        prometheus.io/path: "/metrics"
+    spec:
+      automountServiceAccountToken: false
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 10000
+        runAsGroup: 10000
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: app
+          image: registry.example.com/order-service:v1.8.3
+          ports:
+            - name: http
+              containerPort: 8080
+            - name: metrics
+              containerPort: 9090
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+          resources:
+            requests:
+              cpu: 100m
+              memory: 128Mi
+            limits:
+              memory: 256Mi
+```
+
+### ServiceMonitor (prometheus-operator)
+
+When using prometheus-operator, prefer ServiceMonitor CRDs over annotations for type-safe configuration:
+
+```yaml
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: order-service
+  namespace: orders
+  labels:
+    release: kube-prometheus-stack    # must match Prometheus operator selector
+spec:
+  selector:
+    matchLabels:
+      app: order-service
+  endpoints:
+    - port: metrics                   # must match Service port name
+      interval: 30s
+      path: /metrics
+```
+
+PodMonitor follows the same pattern but targets pods directly (useful when no Service exists, e.g., CronJobs with metrics).
+
+---
+
+## Key Metrics -- the RED Method
+
+Every service should expose at minimum:
+
+| Signal | Metric | Example |
+|---|---|---|
+| **R**ate | Request throughput | `http_requests_total` (counter) |
+| **E**rrors | Failed request count | `http_requests_total{status=~"5.."}` or a dedicated error counter |
+| **D**uration | Request latency | `http_request_duration_seconds` (histogram with buckets) |
+
+For resource-oriented services (queues, databases), add **saturation** metrics: queue depth, connection pool usage, disk I/O utilization.
+
+Use histogram buckets aligned to your SLOs:
+
+```
+http_request_duration_seconds_bucket{le="0.05"}   # 50ms - fast API
+http_request_duration_seconds_bucket{le="0.1"}
+http_request_duration_seconds_bucket{le="0.25"}
+http_request_duration_seconds_bucket{le="0.5"}
+http_request_duration_seconds_bucket{le="1.0"}
+http_request_duration_seconds_bucket{le="2.5"}
+http_request_duration_seconds_bucket{le="5.0"}
+http_request_duration_seconds_bucket{le="10.0"}
+http_request_duration_seconds_bucket{le="+Inf"}
+```
+
+---
+
+## Logging
+
+### Structured JSON to stdout
+
+Applications MUST log structured JSON to stdout/stderr. Never log to files inside the container -- it defeats node-level collection and fills the writable layer.
+
+```json
+{"timestamp":"2025-03-15T10:23:45Z","level":"error","msg":"payment failed","trace_id":"abc123","order_id":"ord-789","error":"timeout after 5s"}
+```
+
+Rules:
+- Use `timestamp`, `level`, `msg` as standard fields.
+- Include `trace_id` and `span_id` for correlation with distributed traces.
+- Never log secrets, tokens, PII, or full request bodies.
+- Use `stderr` for error-level logs and `stdout` for everything else (some collectors distinguish).
+
+### Log aggregation -- DaemonSet pattern
+
+Fluent Bit runs as a DaemonSet on every node, reads container logs from `/var/log/containers/`, and forwards to a sink:
+
+```yaml
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: fluent-bit
+  namespace: logging
+spec:
+  selector:
+    matchLabels:
+      app: fluent-bit
+  template:
+    metadata:
+      labels:
+        app: fluent-bit
+    spec:
+      serviceAccountName: fluent-bit
+      containers:
+        - name: fluent-bit
+          image: fluent/fluent-bit:3.0
+          volumeMounts:
+            - name: varlog
+              mountPath: /var/log
+              readOnly: true
+            - name: containers
+              mountPath: /var/lib/docker/containers
+              readOnly: true
+          resources:
+            requests:
+              cpu: 50m
+              memory: 64Mi
+            limits:
+              memory: 128Mi
+      volumes:
+        - name: varlog
+          hostPath:
+            path: /var/log
+        - name: containers
+          hostPath:
+            path: /var/lib/docker/containers
+```
+
+Node-level collection (DaemonSet) is preferred over sidecar collection for most workloads. Use sidecars only when you need per-pod log transformation or the application cannot log to stdout.
+
+---
+
+## Distributed Tracing -- OpenTelemetry
+
+### Auto-instrumentation with the OTel Operator
+
+The OpenTelemetry Operator can inject instrumentation sidecars via annotation:
+
+```yaml
+metadata:
+  annotations:
+    instrumentation.opentelemetry.io/inject-java: "true"    # or inject-python, inject-nodejs
+```
+
+### OTel Collector sidecar pattern
+
+For fine-grained control, run the OTel Collector as a sidecar:
+
+```yaml
+- name: otel-collector
+  image: otel/opentelemetry-collector-contrib:0.98.0
+  args: ["--config=/etc/otel/config.yaml"]
+  ports:
+    - containerPort: 4317       # gRPC OTLP receiver
+    - containerPort: 4318       # HTTP OTLP receiver
+  securityContext:
+    allowPrivilegeEscalation: false
+    readOnlyRootFilesystem: true
+    capabilities:
+      drop: ["ALL"]
+  resources:
+    requests:
+      cpu: 50m
+      memory: 64Mi
+    limits:
+      memory: 128Mi
+  volumeMounts:
+    - name: otel-config
+      mountPath: /etc/otel
+```
+
+Propagate trace context (`traceparent` header / W3C Trace Context) across all service boundaries. Without propagation, traces are fragmented and useless.
+
+---
+
+## Alerting -- PrometheusRule
+
+Write symptom-based alerts (what the user experiences), not cause-based alerts (what broke internally):
+
+```yaml
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: order-service-alerts
+  namespace: orders
+spec:
+  groups:
+    - name: order-service.rules
+      rules:
+        - alert: HighErrorRate
+          expr: |
+            sum(rate(http_requests_total{job="order-service",status=~"5.."}[5m]))
+            / sum(rate(http_requests_total{job="order-service"}[5m])) > 0.05
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Order service error rate above 5%"
+            runbook_url: "https://wiki.example.com/runbooks/order-service-errors"
+        - alert: HighLatencyP99
+          expr: |
+            histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{job="order-service"}[5m])) by (le)) > 2.0
+          for: 10m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Order service p99 latency above 2s"
+```
+
+Every alert MUST have a `runbook_url` annotation pointing to actionable remediation steps.
+
+---
+
+## Deployment Annotations for Grafana
+
+Annotate deployments in Grafana to correlate metric changes with releases:
+
+```bash
+curl -s -X POST http://grafana.monitoring.svc:3000/api/annotations \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer $GRAFANA_API_KEY" \
+  -d "{\"text\":\"Deployed order-service v1.8.3\",\"tags\":[\"deployment\",\"orders\"]}"
+```
+
+Integrate this into your CI/CD pipeline as a post-deploy step.
+
+---
+
+## LLM Mistake Checklist
+
+Before finalizing any workload manifest or observability configuration, verify each item:
+
+- [ ] **Prometheus annotations** are on the Pod template `metadata.annotations`, not on the Deployment metadata.
+- [ ] **Metrics port** is declared in the container `ports` list and matches the annotation value.
+- [ ] **Logs are structured JSON to stdout** -- no file-based logging, no unstructured text.
+- [ ] **Trace context propagation** is configured -- auto-instrumentation annotation or SDK integration present.
+- [ ] **Alerts are symptom-based** (error rate, latency) not cause-based (pod restarted, CPU high).
+- [ ] **Every alert has a `runbook_url`** annotation -- alerts without runbooks are noise.
+- [ ] **Histogram buckets** are aligned to SLO thresholds, not arbitrary defaults.
+- [ ] **Resource requests and limits** are set on all sidecar containers (OTel Collector, Fluent Bit) to prevent them from starving the main workload.

package/skills/deploy-k8s/references/privilege-sprawl.md

@@ -0,0 +1,273 @@
+# Privilege Sprawl
+
+**Directive:** When generating RBAC resources, ServiceAccounts, or secret references, ALWAYS apply least-privilege principles. Default security posture is PSS "restricted" profile. Never grant permissions broader than the workload requires.
+
+## When to use
+
+Consult this reference whenever the task involves:
+- Creating or modifying Roles, ClusterRoles, RoleBindings, or ClusterRoleBindings
+- Creating ServiceAccounts or referencing them in pod specs
+- Mounting or referencing Kubernetes Secrets
+- Designing access patterns for controllers, operators, or application workloads
+
+## Symptoms of privilege sprawl
+
+| Symptom | Risk |
+|---|---|
+| ClusterRoleBinding with `cluster-admin` attached to a workload SA | Full cluster takeover if pod is compromised |
+| Rules containing `verbs: ["*"]` or `resources: ["*"]` | Unrestricted access far beyond what the workload needs |
+| Pods running with the `default` ServiceAccount | Every pod in the namespace shares the same identity |
+| `automountServiceAccountToken: true` (the default) on pods that never call the API | Leaked token exposes unnecessary attack surface |
+| Secrets injected as environment variables | Visible in `kubectl describe pod`, process listings, crash dumps |
+| Team assumes base64-encoded Secrets are encrypted | Secrets stored in plaintext in etcd unless encryption-at-rest is configured |
+
+## Root causes
+
+1. Copy-pasting cluster-admin bindings from quickstart guides.
+2. Using wildcards to "get it working" and never scoping down.
+3. Not creating dedicated ServiceAccounts per workload.
+4. Misunderstanding that Kubernetes Secrets are base64-encoded, NOT encrypted.
+5. Injecting secrets via `env:` instead of volume mounts or external operators.
+
+## Prevention rules
+
+### RBAC least privilege
+
+- **Role** is namespace-scoped. **ClusterRole** is cluster-scoped. Prefer Role unless access truly spans namespaces.
+- **RoleBinding** binds a Role (or ClusterRole) within a single namespace. **ClusterRoleBinding** grants access cluster-wide.
+- Never bind `cluster-admin` to any workload ServiceAccount. Reserve it for break-glass human access only.
+- List specific verbs: `get`, `list`, `watch`, `create`, `update`, `patch`, `delete`. Never use `"*"`.
+- List specific resources: `pods`, `deployments`, `configmaps`, etc. Never use `"*"`.
+- Always specify `apiGroups` explicitly. An empty string `""` means core API group, not "all groups."
+
+### ServiceAccount hardening
+
+- Create a dedicated ServiceAccount for every workload that needs API access.
+- Set `automountServiceAccountToken: false` on the ServiceAccount AND the Pod spec for workloads that do not call the Kubernetes API.
+- Use projected token volumes with audience and expiration for workloads that do need API access.
+
+### Secret management
+
+- Kubernetes Secrets are base64-encoded, NOT encrypted. Anyone with `get secrets` RBAC in the namespace can read them.
+- Enable etcd encryption at rest via `EncryptionConfiguration` as a baseline.
+- Prefer external secret management: `external-secrets-operator` syncing from AWS Secrets Manager, GCP Secret Manager, or HashiCorp Vault.
+- `sealed-secrets` is an alternative: encrypt secrets client-side so they are safe to commit to git.
+- Mount secrets as files (`volumeMounts`), not environment variables. File-mounted secrets can be rotated without pod restart and are not exposed in `kubectl describe`.
+
+## Patterns and examples
+
+### GOOD: Scoped RBAC + dedicated ServiceAccount + external secrets
+
+```yaml
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: order-processor
+  namespace: orders
+  annotations:
+    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/order-processor
+automountServiceAccountToken: false
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: order-processor-role
+  namespace: orders
+rules:
+  - apiGroups: [""]
+    resources: ["configmaps"]
+    verbs: ["get", "watch"]
+    resourceNames: ["order-config"]
+  - apiGroups: [""]
+    resources: ["events"]
+    verbs: ["create"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: order-processor-binding
+  namespace: orders
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: order-processor-role
+subjects:
+  - kind: ServiceAccount
+    name: order-processor
+    namespace: orders
+---
+apiVersion: external-secrets.io/v1beta1
+kind: ExternalSecret
+metadata:
+  name: order-db-creds
+  namespace: orders
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    name: aws-secrets-manager
+    kind: ClusterSecretStore
+  target:
+    name: order-db-creds
+  data:
+    - secretKey: password
+      remoteRef:
+        key: prod/orders/db-password
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: order-processor
+  namespace: orders
+spec:
+  replicas: 3
+  selector:
+    matchLabels:
+      app: order-processor
+  template:
+    metadata:
+      labels:
+        app: order-processor
+    spec:
+      serviceAccountName: order-processor
+      automountServiceAccountToken: false
+      securityContext:
+        runAsNonRoot: true
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: processor
+          image: registry.example.com/order-processor:v2.4.1
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]
+            readOnlyRootFilesystem: true
+          volumeMounts:
+            - name: db-creds
+              mountPath: /etc/secrets/db
+              readOnly: true
+      volumes:
+        - name: db-creds
+          secret:
+            secretName: order-db-creds
+```
+
+### BAD: cluster-admin binding + default SA + env var secrets
+
+```yaml
+# DO NOT DO THIS
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: order-processor-admin
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: cluster-admin          # grants full cluster control to a workload
+subjects:
+  - kind: ServiceAccount
+    name: default              # shared by every pod in the namespace
+    namespace: orders
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: order-processor
+  namespace: orders
+spec:
+  replicas: 3
+  selector:
+    matchLabels:
+      app: order-processor
+  template:
+    metadata:
+      labels:
+        app: order-processor
+    spec:
+      # serviceAccountName omitted -- uses "default"
+      # automountServiceAccountToken defaults to true -- token exposed
+      containers:
+        - name: processor
+          image: registry.example.com/order-processor:latest
+          env:
+            - name: DB_PASSWORD                   # visible in describe, logs, crash dumps
+              value: "hunter2"                     # hardcoded plaintext password
+            - name: DB_PASSWORD_FROM_SECRET
+              valueFrom:
+                secretKeyRef:
+                  name: db-creds
+                  key: password                    # still exposed via env, not file mount
+```
+
+### Token projection for workloads that need API access
+
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+  name: api-consumer
+  namespace: orders
+spec:
+  serviceAccountName: order-processor
+  automountServiceAccountToken: false   # disable the default mount
+  containers:
+    - name: app
+      image: registry.example.com/api-consumer:v1.0.0
+      volumeMounts:
+        - name: kube-api-token
+          mountPath: /var/run/secrets/tokens
+          readOnly: true
+  volumes:
+    - name: kube-api-token
+      projected:
+        sources:
+          - serviceAccountToken:
+              audience: "https://kubernetes.default.svc"
+              expirationSeconds: 3600
+              path: token
+```
+
+## LLM mistake checklist
+
+Before finalizing any RBAC or secret-related manifest, verify each item:
+
+- [ ] No rule uses `verbs: ["*"]` -- every verb is listed explicitly
+- [ ] No rule uses `resources: ["*"]` -- every resource is listed explicitly
+- [ ] No rule uses `apiGroups: ["*"]` -- each API group is listed explicitly
+- [ ] No ClusterRoleBinding references `cluster-admin` for a workload ServiceAccount
+- [ ] A dedicated ServiceAccount is created (not relying on `default`)
+- [ ] `automountServiceAccountToken: false` is set on pods that do not need API access
+- [ ] Secrets are mounted as volumes, not injected as environment variables
+- [ ] No hardcoded secret values appear in the manifest (use ExternalSecret, SealedSecret, or at minimum a Secret resource)
+- [ ] `resourceNames` is used where possible to restrict access to specific named resources
+- [ ] RoleBinding is preferred over ClusterRoleBinding unless cluster-wide scope is required
+- [ ] Pod securityContext sets `runAsNonRoot: true`, drops all capabilities, enables seccomp
+
+## Verification commands
+
+```bash
+# Check what a specific ServiceAccount can do
+kubectl auth can-i --list --as=system:serviceaccount:orders:order-processor -n orders
+
+# Check if a ServiceAccount can perform a specific action
+kubectl auth can-i get secrets --as=system:serviceaccount:orders:order-processor -n orders
+
+# Find all ClusterRoleBindings that reference cluster-admin
+kubectl get clusterrolebindings -o json | \
+  jq -r '.items[] | select(.roleRef.name=="cluster-admin") | .metadata.name + " -> " + (.subjects[]? | .kind + "/" + .name)'
+
+# Find RBAC rules with wildcard verbs or resources
+kubectl get roles,clusterroles -A -o json | \
+  jq -r '.items[] | select(.rules[]? | .verbs[]? == "*" or .resources[]? == "*") | .metadata.namespace + "/" + .metadata.name'
+
+# List all pods using the default ServiceAccount
+kubectl get pods -A -o json | \
+  jq -r '.items[] | select(.spec.serviceAccountName == "default" or .spec.serviceAccountName == null) | .metadata.namespace + "/" + .metadata.name'
+
+# Check if etcd encryption at rest is enabled (control plane access required)
+kubectl get apiserver -o=jsonpath='{.items[0].spec.encryption}'
+
+# Audit secrets exposed as environment variables
+kubectl get pods -A -o json | \
+  jq -r '.items[] | .metadata.namespace + "/" + .metadata.name as $pod | .spec.containers[]?.env[]? | select(.valueFrom.secretKeyRef != null) | $pod + " env:" + .name'
+```