@jetrabbits/agentic 0.0.1

package/areas/devops/kubernetes/skills/rbac-design/SKILL.md

@@ -0,0 +1,148 @@
+---
+name: rbac-design
+type: skill
+description: Design minimal-privilege RBAC for workloads, operators, and human access in multi-tenant clusters.
+related-rules:
+  - workload-security.md
+allowed-tools: Read, Write, Edit, Bash
+---
+
+# Skill: RBAC Design
+
+> **Expertise:** Kubernetes RBAC — service accounts, Roles, ClusterRoles, namespace isolation, human access patterns.
+
+## When to load
+
+When onboarding a new service, setting up CI/CD cluster access, auditing permissions, or debugging "forbidden" API errors.
+
+## RBAC Object Hierarchy
+
+```
+ClusterRole         → cluster-scoped permissions (nodes, PVs, namespaces)
+Role                → namespace-scoped permissions (pods, services, configmaps)
+ClusterRoleBinding  → binds ClusterRole to subject cluster-wide
+RoleBinding         → binds Role OR ClusterRole to subject in one namespace
+```
+
+## Workload Service Account Pattern
+
+```yaml
+# 1. Dedicated ServiceAccount per workload
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: order-service
+  namespace: production
+  annotations:
+    # For cloud IAM federation (AWS IRSA, GCP Workload Identity)
+    eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/order-service-prod
+automountServiceAccountToken: false  # disable unless needed
+
+---
+# 2. Role — minimal permissions
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: order-service
+  namespace: production
+rules:
+  - apiGroups: [""]
+    resources: ["configmaps"]
+    verbs: ["get", "list", "watch"]
+    resourceNames: ["order-service-config"]  # scope to specific resource
+  - apiGroups: [""]
+    resources: ["secrets"]
+    verbs: ["get"]
+    resourceNames: ["order-service-tls"]
+
+---
+# 3. RoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: order-service
+  namespace: production
+subjects:
+  - kind: ServiceAccount
+    name: order-service
+    namespace: production
+roleRef:
+  kind: Role
+  apiGroupv: rbac.authorization.k8s.io
+  name: order-service
+```
+
+## Human Access Patterns
+
+```yaml
+# Dev read-only access to staging namespace
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: devs-view-staging
+  namespace: staging
+subjects:
+  - kind: Group
+    name: developers           # from OIDC provider (Dex, Okta, etc.)
+    apiGroup: rbac.authorization.k8s.io
+roleRef:
+  kind: ClusterRole
+  name: view                   # built-in read-only ClusterRole
+  apiGroup: rbac.authorization.k8s.io
+```
+
+## Built-in ClusterRoles (use before creating custom)
+
+| ClusterRole | Access level |
+|:---|:---|
+| `view` | Read-only all namespaced resources |
+| `edit` | Read/write most namespaced resources; no RBAC |
+| `admin` | Full namespace access including RBAC |
+| `cluster-admin` | Full cluster access — **never bind to apps** |
+
+## CI/CD Access Pattern
+
+```yaml
+# CI system gets minimal cluster access
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: ci-deployer
+rules:
+  - apiGroups: ["apps"]
+    resources: ["deployments", "statefulsets"]
+    verbs: ["get", "list", "patch", "update"]
+  - apiGroups: [""]
+    resources: ["pods"]
+    verbs: ["get", "list"]
+  # NOT: create/delete pods, access secrets, modify RBAC
+```
+
+## RBAC Audit Commands
+
+```bash
+# What can a ServiceAccount do?
+kubectl auth can-i --list \
+  --as=system:serviceaccount:production:order-service \
+  -n production
+
+# Who can do X in namespace Y?
+kubectl who-can get secrets -n production    # requires kubectl-who-can plugin
+
+# Find all RoleBindings in a namespace
+kubectl get rolebindings,clusterrolebindings -n production -o wide
+
+# Check if a specific action is allowed
+kubectl auth can-i delete pods -n production \
+  --as=system:serviceaccount:production:order-service
+```
+
+## Common Misconfigurations
+
+| Mistake | Risk | Fix |
+|:---|:---|:---|
+| Using `default` ServiceAccount | All pods in namespace share permissions | Dedicate one SA per workload |
+| `verbs: ["*"]` | Full resource control | Enumerate exact verbs needed |
+| `resources: ["*"]` | Access to all resources | List explicitly |
+| Binding `cluster-admin` to CI | Breach = full cluster takeover | Use scoped `ci-deployer` ClusterRole |
+| `automountServiceAccountToken: true` (default) | Token injected into all pods | Set to `false` unless needed |

package/areas/devops/kubernetes/skills/resource-tuning/SKILL.md

@@ -0,0 +1,156 @@
+---
+name: resource-tuning
+type: skill
+description: Right-size pod resources, configure HPA/VPA/KEDA, and eliminate resource waste in Kubernetes.
+related-rules:
+  - resource-governance.md
+allowed-tools: Read, Bash
+---
+
+# Skill: Resource Tuning
+
+> **Expertise:** CPU/memory right-sizing, HPA, VPA, KEDA event-driven scaling, namespace quota design.
+
+## When to load
+
+When pods are OOMKilled, CPU-throttled, underutilised, or autoscaling isn't working as expected.
+
+## Right-Sizing Methodology
+
+```
+1. Observe → 7-day peak metrics (kubectl top / Prometheus)
+2. Set request = average × 1.1 (room for normal variance)
+3. Set limit = p99 peak × 1.3 (room for spike without OOM)
+4. Verify no throttling with: throttled_cpu_seconds metric
+5. Adjust after 2 weeks of production data
+```
+
+```bash
+# Current resource usage (snapshot)
+kubectl top pods -n <ns> --sort-by=memory
+kubectl top pods -n <ns> --sort-by=cpu
+
+# Historical usage via Prometheus (7-day p99)
+# CPU p99:
+rate(container_cpu_usage_seconds_total{namespace="<ns>",pod=~"my-app-.*"}[5m])
+  > quantile_over_time(0.99, rate(...[5m])[7d:5m])
+
+# Memory p99:
+quantile_over_time(0.99,
+  container_memory_working_set_bytes{namespace="<ns>",pod=~"my-app-.*"}[7d:5m])
+```
+
+## HPA Configuration
+
+```yaml
+# CPU + Memory HPA (Kubernetes 1.23+)
+apiVersion: autoscaling/v2
+kind: HorizontalPodAutoscaler
+metadata:
+  name: my-service
+  namespace: production
+spec:
+  scaleTargetRef:
+    apiVersion: apps/v1
+    kind: Deployment
+    name: my-service
+  minReplicas: 2
+  maxReplicas: 20
+  metrics:
+    - type: Resource
+      resource:
+        name: cpu
+        target:
+          type: Utilization
+          averageUtilization: 70    # scale out at 70% avg CPU
+    - type: Resource
+      resource:
+        name: memory
+        target:
+          type: AverageValue
+          averageValue: 400Mi       # scale out if avg pod memory > 400Mi
+  behavior:
+    scaleDown:
+      stabilizationWindowSeconds: 300   # wait 5 min before scaling down
+      policies:
+        - type: Pods
+          value: 1
+          periodSeconds: 60             # scale down max 1 pod per minute
+    scaleUp:
+      stabilizationWindowSeconds: 0    # scale up immediately
+      policies:
+        - type: Percent
+          value: 100
+          periodSeconds: 15            # can double every 15 seconds
+```
+
+## VPA (Vertical Pod Autoscaler)
+
+```yaml
+# VPA in "Off" mode — recommendations only, no auto-apply
+apiVersion: autoscaling.k8s.io/v1
+kind: VerticalPodAutoscaler
+metadata:
+  name: my-service-vpa
+  namespace: production
+spec:
+  targetRef:
+    apiVersion: apps/v1
+    kind: Deployment
+    name: my-service
+  updatePolicy:
+    updateMode: "Off"    # "Auto" restarts pods — risky in prod; use "Off" first
+  resourcePolicy:
+    containerPolicies:
+      - containerName: "*"
+        minAllowed: { cpu: 50m, memory: 64Mi }
+        maxAllowed: { cpu: 2,   memory: 2Gi }
+
+# Check VPA recommendations
+kubectl describe vpa my-service-vpa -n production | grep -A20 "Recommendation:"
+```
+
+## KEDA (Event-Driven Autoscaling)
+
+```yaml
+# Scale based on RabbitMQ queue depth
+apiVersion: keda.sh/v1alpha1
+kind: ScaledObject
+metadata:
+  name: worker-scaledobject
+  namespace: production
+spec:
+  scaleTargetRef:
+    name: task-worker
+  minReplicaCount: 1
+  maxReplicaCount: 30
+  cooldownPeriod: 60
+  triggers:
+    - type: rabbitmq
+      metadata:
+        host: amqp://rabbitmq.infra.svc.cluster.local:5672
+        queueName: task-queue
+        queueLength: "20"    # 1 replica per 20 messages in queue
+```
+
+## CPU Throttling Detection
+
+```bash
+# Check CPU throttling in Prometheus
+# > 25% throttling indicates limit is too low
+100 * sum(rate(container_cpu_throttled_seconds_total{
+  namespace="<ns>", container!=""}[5m]))
+/ sum(rate(container_cpu_usage_seconds_total{
+  namespace="<ns>", container!=""}[5m]))
+
+# Quick check per pod
+kubectl exec -it <pod> -n <ns> -- cat /sys/fs/cgroup/cpu/cpu.stat | grep throttled
+```
+
+## ResourceQuota Design by Team Size
+
+| Team size | CPU quota | Memory quota | Pod count |
+|:---|:---|:---|:---|
+| 1–3 services | 8 cores | 16Gi | 50 |
+| 4–10 services | 20 cores | 40Gi | 150 |
+| >10 services | per-capacity-plan | per-capacity-plan | 300+ |

package/areas/devops/kubernetes/workflows/cluster-bootstrap.md

@@ -0,0 +1,194 @@
+---
+name: cluster-bootstrap
+type: workflow
+trigger: /cluster-bootstrap
+description: Bootstrap a production-grade self-hosted Kubernetes cluster from bare-metal using kubeadm, with Cilium CNI, MetalLB, ArgoCD, and cert-manager.
+inputs:
+  - node_inventory (list of IPs/hostnames)
+  - cluster_name
+  - pod_cidr
+  - service_cidr
+outputs:
+  - running_cluster
+  - kubeconfig
+  - bootstrap_report
+roles:
+  - devops-engineer
+execution:
+  initiator: developer
+related-rules:
+  - cluster-standards.md
+  - workload-security.md
+uses-skills:
+  - cluster-operations
+  - helm-charts
+quality-gates:
+  - all nodes pass pre-flight checks
+  - control plane HA (3 nodes) before adding workers
+  - etcd encrypted at rest
+  - CNI healthy before any workloads deployed
+---
+
+## Steps
+
+### 1. Node Pre-Flight — `@devops-engineer`
+- **Actions (all nodes via Ansible or manual):**
+  ```bash
+  # Disable swap (K8s requirement)
+  swapoff -a && sed -i '/swap/d' /etc/fstab
+
+  # Load required kernel modules
+  cat > /etc/modules-load.d/k8s.conf << EOF
+  overlay
+  br_netfilter
+  EOF
+  modprobe overlay && modprobe br_netfilter
+
+  # Kernel parameters
+  cat > /etc/sysctl.d/k8s.conf << EOF
+  net.bridge.bridge-nf-call-iptables  = 1
+  net.bridge.bridge-nf-call-ip6tables = 1
+  net.ipv4.ip_forward                 = 1
+  EOF
+  sysctl --system
+
+  # Install containerd
+  apt-get install -y containerd
+  mkdir -p /etc/containerd
+  containerd config default > /etc/containerd/config.toml
+  # Enable SystemdCgroup in containerd config
+  sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
+  systemctl restart containerd
+
+  # Install kubeadm, kubelet, kubectl (pin version)
+  apt-get install -y kubeadm=1.31.* kubelet=1.31.* kubectl=1.31.*
+  apt-mark hold kubeadm kubelet kubectl
+  ```
+- **Done when:** all nodes pass `kubeadm init phase preflight`
+
+### 2. Bootstrap First Control Plane — `@devops-engineer`
+- **Input:** cluster_name, pod_cidr, service_cidr, VIP for HA (keepalived/haproxy)
+- **Actions:**
+  ```bash
+  # kubeadm config file (preferred over flags)
+  cat > kubeadm-config.yaml << EOF
+  apiVersion: kubeadm.k8s.io/v1beta3
+  kind: ClusterConfiguration
+  clusterName: ${CLUSTER_NAME}
+  controlPlaneEndpoint: "${VIP}:6443"   # HA VIP
+  networking:
+    podSubnet: "${POD_CIDR}"            # e.g. 10.244.0.0/16
+    serviceSubnet: "${SVC_CIDR}"        # e.g. 10.96.0.0/12
+  ---
+  apiVersion: kubeadm.k8s.io/v1beta3
+  kind: InitConfiguration
+  nodeRegistration:
+    criSocket: unix:///run/containerd/containerd.sock
+  EOF
+
+  kubeadm init --config kubeadm-config.yaml --upload-certs
+
+  # Configure kubectl
+  mkdir -p ~/.kube
+  cp /etc/kubernetes/admin.conf ~/.kube/config
+  ```
+- **Done when:** `kubectl get nodes` shows first control plane node (NotReady — CNI not yet installed)
+
+### 3. Install CNI (Cilium) — `@devops-engineer`
+- **Actions:**
+  ```bash
+  # Install Cilium CLI
+  cilium install \
+    --set ipam.mode=kubernetes \
+    --set kubeProxyReplacement=true \
+    --set hubble.enabled=true \
+    --set hubble.relay.enabled=true \
+    --set hubble.ui.enabled=true
+
+  # Verify
+  cilium status --wait
+  ```
+- **Done when:** `kubectl get nodes` shows control plane `Ready`; `cilium status` shows OK
+
+### 4. Join Remaining Control Plane Nodes — `@devops-engineer`
+- **Actions (on each additional CP node):**
+  ```bash
+  # Use join command from `kubeadm init` output (includes --control-plane --certificate-key)
+  kubeadm join ${VIP}:6443 \
+    --token <token> \
+    --discovery-token-ca-cert-hash sha256:<hash> \
+    --control-plane \
+    --certificate-key <cert-key>
+  ```
+- **Done when:** `kubectl get nodes` shows 3 control plane nodes `Ready`
+
+### 5. Join Worker Nodes — `@devops-engineer`
+- **Actions (on each worker):**
+  ```bash
+  kubeadm join ${VIP}:6443 \
+    --token <token> \
+    --discovery-token-ca-cert-hash sha256:<hash>
+  ```
+  - Label workers: `kubectl label node <n> node-role.kubernetes.io/worker= topology.kubernetes.io/zone=<zone>`
+- **Done when:** all workers `Ready` in `kubectl get nodes`
+
+### 6. etcd Encryption at Rest — `@devops-engineer`
+- **Actions:**
+  ```bash
+  # Create EncryptionConfiguration
+  cat > /etc/kubernetes/enc/encryption-config.yaml << EOF
+  apiVersion: apiserver.config.k8s.io/v1
+  kind: EncryptionConfiguration
+  resources:
+    - resources: [secrets, configmaps]
+      providers:
+        - aescbc:
+            keys:
+              - name: key1
+                secret: $(head -c 32 /dev/urandom | base64)
+        - identity: {}
+  EOF
+
+  # Add to kube-apiserver static pod manifest:
+  # --encryption-provider-config=/etc/kubernetes/enc/encryption-config.yaml
+
+  # Re-encrypt all existing secrets
+  kubectl get secrets -A -o json | kubectl replace -f -
+  ```
+
+### 7. Core Platform Components — `@devops-engineer`
+- **Install in order:**
+  ```bash
+  # MetalLB (bare-metal load balancer)
+  helm upgrade --install metallb metallb/metallb -n metallb-system --create-namespace
+  # Apply IPAddressPool with your bare-metal IP range
+
+  # cert-manager
+  helm upgrade --install cert-manager jetstack/cert-manager \
+    -n cert-manager --create-namespace \
+    --set installCRDs=true
+
+  # NGINX Ingress Controller
+  helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \
+    -n ingress-nginx --create-namespace
+
+  # ArgoCD
+  helm upgrade --install argocd argo/argo-cd \
+    -n argocd --create-namespace \
+    -f infra/argocd/values.yaml
+
+  # External Secrets Operator
+  helm upgrade --install external-secrets external-secrets/external-secrets \
+    -n external-secrets --create-namespace
+  ```
+
+### 8. Apply Security Baselines — `@devops-engineer`
+- **Actions:**
+  - Apply OPA/Gatekeeper or Kyverno policies from `infra/policies/`
+  - Create default namespace deny-all NetworkPolicy template
+  - Configure etcd backup CronJob
+  - Set up `kube-prometheus-stack` for cluster monitoring
+- **Output:** `bootstrap_report.md` — cluster version, node IPs, installed components, kubeconfig location
+
+## Exit
+All nodes Ready + core components Running + etcd encrypted + monitoring live = cluster bootstrapped.

package/areas/devops/kubernetes/workflows/debug-workload.md

@@ -0,0 +1,108 @@
+---
+name: debug-workload
+type: workflow
+trigger: /debug-workload
+description: Systematic diagnosis of Kubernetes workload failures — from symptom to root cause and fix.
+inputs:
+  - workload_name
+  - namespace
+  - symptom_description
+outputs:
+  - root_cause_summary
+  - remediation_applied
+roles:
+  - developer
+  - devops-engineer
+execution:
+  initiator: developer
+related-rules:
+  - workload-security.md
+  - resource-governance.md
+uses-skills:
+  - pod-troubleshooting
+  - cluster-operations
+quality-gates:
+  - symptom reproduced before fix applied
+  - fix verified in target environment
+  - root cause documented
+---
+
+## Steps
+
+### 1. Classify Symptom — `@devops-engineer`
+- **Input:** workload name, namespace, symptom description
+- **Actions:**
+  - `kubectl get pods -n <ns> -l app=<name> -o wide` — check pod statuses
+  - Classify into: Scheduling issue / Runtime crash / OOM / ImagePull / Service unreachable
+  - Check recent events: `kubectl get events -n <ns> --sort-by='.lastTimestamp' | tail -20`
+- **Output:** symptom classification (one of the above categories)
+- **Done when:** root failure mode identified at pod level
+
+### 2. Deep Diagnosis — `@devops-engineer`
+- **Input:** symptom classification
+- **Actions by category:**
+
+  **CrashLoopBackOff:**
+  ```bash
+  kubectl logs <pod> -n <ns> --previous --tail=200
+  kubectl describe pod <pod> -n <ns> | grep -A3 "Last State:"
+  # Exit 137 = OOMKilled → raise memory limit
+  # Exit 1 = app error → read logs
+  # Exit 143 = SIGTERM → fix graceful shutdown
+  ```
+
+  **Pending:**
+  ```bash
+  kubectl describe pod <pod> -n <ns> | grep -A20 "Events:"
+  kubectl describe nodes | grep -A5 "Allocated resources:"
+  kubectl get nodes --show-labels | grep -v NotReady
+  ```
+
+  **ImagePullBackOff:**
+  ```bash
+  kubectl describe pod <pod> -n <ns> | grep -A5 "Failed to pull"
+  kubectl get secret regcred -n <ns> -o yaml  # verify imagePullSecret
+  ```
+
+  **Service unreachable:**
+  ```bash
+  kubectl get endpoints <svc> -n <ns>   # empty = label selector mismatch
+  kubectl describe svc <svc> -n <ns>    # check selector labels
+  # Test DNS: kubectl run test --image=busybox -it --rm -- nslookup <svc>.<ns>
+  ```
+
+- **Output:** root cause identified with evidence
+- **Done when:** can explain exactly why the workload failed
+
+### 3. Apply Fix — `@developer` + `@devops-engineer`
+- **Input:** root cause
+- **Actions:**
+  - Fix via Helm values / manifest change (never `kubectl edit` directly in production)
+  - Commit change to Git; deploy via ArgoCD or CI pipeline
+  - For P0/P1 only: `kubectl patch` as emergency measure + follow-up Git change within 1 hour
+- **Output:** fixed manifest merged to Git + applied to cluster
+- **Done when:** pods enter `Running` state; readiness probe passing
+
+### 4. Verify & Monitor — `@devops-engineer`
+- **Input:** deployed fix
+- **Actions:**
+  ```bash
+  kubectl rollout status deployment/<name> -n <ns>
+  kubectl get pods -n <ns> -l app=<name> -w   # watch for 2 minutes
+  kubectl logs -n <ns> -l app=<name> --tail=50 # confirm no new errors
+  ```
+  - Check relevant Grafana dashboard for error rate and latency
+- **Output:** workload healthy confirmation
+- **Done when:** all pods `Running`, metrics normal, no log errors for 5 minutes
+
+### 5. Document — `@devops-engineer`
+- **Input:** root cause + fix applied
+- **Actions:** write brief `root_cause_summary.md`:
+  - What failed, why, which resource/manifest was at fault
+  - Fix applied (link to commit)
+  - Prevention: add to runbook? Add Prometheus alert? Change default values?
+- **Output:** `docs/incidents/<date>-<workload>-root-cause.md`
+- **Done when:** document committed; alert/runbook created if pattern is recurring
+
+## Exit
+Pod Running + metrics stable + root cause documented = workload debug complete.