@jetrabbits/agentic 0.0.1

package/areas/devops/kubernetes/skills/cluster-operations/SKILL.md

@@ -0,0 +1,136 @@
+---
+name: cluster-operations
+type: skill
+description: Day-2 cluster operations — node management, etcd backup/restore, certificate rotation, namespace lifecycle.
+related-rules:
+  - cluster-standards.md
+  - upgrade-policy.md
+allowed-tools: Read, Bash
+---
+
+# Skill: Cluster Operations
+
+> **Expertise:** Safe day-2 operations on self-hosted Kubernetes clusters — node drain, etcd ops, cert rotation.
+
+## When to load
+
+When draining nodes for maintenance, rotating certificates, backing up etcd, or troubleshooting control plane issues.
+
+## Node Lifecycle Operations
+
+```bash
+# --- CORDON (stop scheduling new pods, don't evict existing) ---
+kubectl cordon <node-name>
+# Use case: pre-drain notification, temporary maintenance hold
+
+# --- DRAIN (evict all pods, mark unschedulable) ---
+kubectl drain <node-name> \
+  --ignore-daemonsets \          # DaemonSet pods can't be evicted
+  --delete-emptydir-data \       # required for pods using emptyDir
+  --grace-period=60 \            # give pods time to shut down cleanly
+  --timeout=300s                 # abort if takes > 5 minutes
+# After drain: node is unschedulable and empty (except daemonsets)
+
+# --- UNCORDON (return to service) ---
+kubectl uncordon <node-name>
+
+# --- Verify node is empty before maintenance ---
+kubectl get pods -A --field-selector=spec.nodeName=<node-name>
+```
+
+## etcd Backup (bare-metal / kubeadm)
+
+```bash
+# --- Take snapshot (run on a control plane node) ---
+ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%Y%m%d-%H%M%S).db \
+  --endpoints=https://127.0.0.1:2379 \
+  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
+  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
+  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key
+
+# --- Verify snapshot ---
+ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-latest.db --write-out=table
+
+# --- Restore snapshot (disaster recovery — only when cluster is down) ---
+ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-latest.db \
+  --data-dir=/var/lib/etcd-restored \
+  --initial-cluster=master-1=https://192.168.1.10:2380 \
+  --initial-advertise-peer-urls=https://192.168.1.10:2380 \
+  --name=master-1
+# Then update etcd static pod manifest to point to new data-dir
+```
+
+## Certificate Rotation (kubeadm)
+
+```bash
+# --- Check certificate expiry ---
+kubeadm certs check-expiration
+
+# --- Renew all certificates (run on each control plane node) ---
+kubeadm certs renew all
+
+# --- Restart control plane components after renewal ---
+# (kubeadm renews certs but doesn't restart static pods automatically)
+for pod in kube-apiserver kube-controller-manager kube-scheduler; do
+  kubectl -n kube-system delete pod -l component=$pod
+done
+
+# --- Update kubeconfig after cert renewal ---
+cp /etc/kubernetes/admin.conf ~/.kube/config
+```
+
+## Namespace Lifecycle
+
+```bash
+# --- Create namespace with standard labels ---
+kubectl create namespace my-team-prod
+kubectl label namespace my-team-prod \
+  environment=production \
+  team=my-team \
+  pod-security.kubernetes.io/enforce=restricted
+
+# --- Apply default NetworkPolicy and LimitRange immediately ---
+kubectl apply -f infra/namespaces/defaults/ -n my-team-prod
+
+# --- Safe namespace deletion (check for resources first) ---
+kubectl get all -n <namespace-to-delete>
+kubectl delete namespace <name>   # blocks until all resources are gone
+# If stuck in Terminating:
+kubectl get namespace <name> -o json | \
+  jq '.spec.finalizers = []' | \
+  kubectl replace --raw "/api/v1/namespaces/<name>/finalize" -f -
+```
+
+## Control Plane Health Checks
+
+```bash
+# API server, scheduler, controller-manager
+kubectl get componentstatuses       # deprecated in 1.19+ but still useful
+kubectl get pods -n kube-system     # all system pods should be Running
+
+# etcd cluster health
+ETCDCTL_API=3 etcdctl endpoint health \
+  --endpoints=https://127.0.0.1:2379 \
+  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
+  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
+  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key
+
+# Node conditions
+kubectl describe nodes | grep -A5 "Conditions:"
+```
+
+## Useful Aliases / One-liners
+
+```bash
+# All pods not Running
+kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded
+
+# Recent events by namespace
+kubectl get events -n <ns> --sort-by='.lastTimestamp'
+
+# Resource usage by namespace
+kubectl top pods -A --sort-by=memory | head -20
+
+# Find pods on a specific node
+kubectl get pods -A -o wide | grep <node-name>
+```

package/areas/devops/kubernetes/skills/helm-charts/SKILL.md

@@ -0,0 +1,152 @@
+---
+name: helm-charts
+type: skill
+description: Design, structure, and test production-grade Helm charts with multi-environment overlays.
+related-rules:
+  - workload-security.md
+  - resource-governance.md
+allowed-tools: Read, Write, Edit, Bash
+---
+
+# Skill: Helm Charts
+
+> **Expertise:** Helm 3 chart structure, values hierarchy, multi-environment overlays, chart testing, ArgoCD integration.
+
+## When to load
+
+When creating a new Helm chart, reviewing an existing chart, setting up multi-env values, or integrating with ArgoCD.
+
+## Chart Structure (Standard)
+
+```
+charts/my-service/
+├── Chart.yaml
+├── values.yaml            ← defaults (all envs inherit)
+├── values-staging.yaml    ← staging overrides
+├── values-prod.yaml       ← production overrides
+├── templates/
+│   ├── _helpers.tpl       ← named templates
+│   ├── deployment.yaml
+│   ├── service.yaml
+│   ├── ingress.yaml
+│   ├── hpa.yaml
+│   ├── pdb.yaml
+│   ├── serviceaccount.yaml
+│   ├── networkpolicy.yaml
+│   └── NOTES.txt
+└── tests/
+    └── test-connection.yaml
+```
+
+## values.yaml Conventions
+
+```yaml
+# Always provide a complete, renderable default set
+replicaCount: 2
+
+image:
+  repository: registry.example.com/my-service
+  tag: ""              # overridden by CI with digest
+  digest: ""           # prefer digest over tag in prod
+  pullPolicy: IfNotPresent
+
+serviceAccount:
+  create: true
+  name: ""             # auto-generated from chart name if empty
+
+resources:
+  requests: { cpu: 100m, memory: 128Mi }
+  limits:   { cpu: 500m, memory: 512Mi }
+
+autoscaling:
+  enabled: true
+  minReplicas: 2
+  maxReplicas: 10
+  targetCPUUtilizationPercentage: 70
+
+ingress:
+  enabled: false
+  className: nginx
+  hosts: []
+  tls: []
+
+podDisruptionBudget:
+  enabled: true
+  minAvailable: 1
+```
+
+## _helpers.tpl Essentials
+
+```yaml
+{{/* Selector labels — must be stable across upgrades */}}
+{{- define "app.selectorLabels" -}}
+app.kubernetes.io/name: {{ include "app.name" . }}
+app.kubernetes.io/instance: {{ .Release.Name }}
+{{- end }}
+
+{{/* Use digest when available, fall back to tag */}}
+{{- define "app.image" -}}
+{{- if .Values.image.digest -}}
+{{ .Values.image.repository }}@{{ .Values.image.digest }}
+{{- else -}}
+{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}
+{{- end -}}
+{{- end }}
+```
+
+## Multi-Environment with ArgoCD
+
+```yaml
+# argocd/apps/my-service-prod.yaml
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: my-service-prod
+  namespace: argocd
+spec:
+  project: production
+  source:
+    repoURL: https://git.example.com/infra/charts
+    targetRevision: HEAD
+    path: charts/my-service
+    helm:
+      valueFiles:
+        - values.yaml
+        - values-prod.yaml
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: my-service-prod
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true
+      - ServerSideApply=true
+```
+
+## Chart Testing
+
+```bash
+# Lint (catches YAML errors + best practice violations)
+helm lint charts/my-service/ -f charts/my-service/values-prod.yaml
+
+# Render and inspect (no cluster needed)
+helm template my-service charts/my-service/ \
+  -f charts/my-service/values-prod.yaml \
+  --debug | kubectl apply --dry-run=client -f -
+
+# Integration test with chart-testing (ct)
+ct lint --chart-dirs charts/ --config ct.yaml
+ct install --chart-dirs charts/ --config ct.yaml   # deploys to kind cluster
+```
+
+## Anti-Patterns
+
+| Anti-pattern | Fix |
+|:---|:---|
+| `image.tag: latest` | Use content digest `image.digest: sha256:...` |
+| Hardcoded namespace in templates | Use `.Release.Namespace` |
+| All config in one values.yaml | Split by environment; prod values in separate file |
+| No `resources` block | Always set requests + limits in values defaults |
+| `helm install` without `--atomic` in CI | Use `--atomic --timeout 5m` for automatic rollback |

package/areas/devops/kubernetes/skills/network-policies/SKILL.md

@@ -0,0 +1,169 @@
+---
+name: network-policies
+type: skill
+description: Design and implement Kubernetes NetworkPolicy and Cilium network policies for namespace isolation and service-to-service access control.
+related-rules:
+  - workload-security.md
+  - cluster-standards.md
+allowed-tools: Read, Write, Edit, Bash
+---
+
+# Skill: Network Policies
+
+> **Expertise:** K8s NetworkPolicy + Cilium policy design for multi-tenant namespace isolation and zero-trust traffic control.
+
+## When to load
+
+When isolating a new namespace, allowing specific service-to-service communication, debugging traffic being blocked, or auditing inter-namespace access.
+
+## Standard Policy Set (apply to every new namespace)
+
+```yaml
+# 1. Default deny-all (must be first)
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: default-deny-all
+  namespace: my-app
+spec:
+  podSelector: {}              # matches ALL pods in namespace
+  policyTypes: [Ingress, Egress]
+
+---
+# 2. Allow DNS (required for all pods)
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: allow-dns-egress
+  namespace: my-app
+spec:
+  podSelector: {}
+  policyTypes: [Egress]
+  egress:
+    - ports:
+        - port: 53
+          protocol: UDP
+        - port: 53
+          protocol: TCP
+
+---
+# 3. Allow ingress from ingress controller
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: allow-ingress-controller
+  namespace: my-app
+spec:
+  podSelector:
+    matchLabels:
+      app: my-service
+  policyTypes: [Ingress]
+  ingress:
+    - from:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: ingress-nginx
+      ports:
+        - port: 8080
+```
+
+## Service-to-Service Policy
+
+```yaml
+# Allow order-service (in orders ns) to call payment-service (in payments ns)
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: allow-from-orders
+  namespace: payments
+spec:
+  podSelector:
+    matchLabels:
+      app: payment-service
+  policyTypes: [Ingress]
+  ingress:
+    - from:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: orders
+          podSelector:
+            matchLabels:
+              app: order-service
+      ports:
+        - port: 8080
+```
+
+## Monitoring Ingress (Prometheus scraping)
+
+```yaml
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: allow-prometheus-scrape
+  namespace: my-app
+spec:
+  podSelector: {}              # allow scraping all pods in ns
+  policyTypes: [Ingress]
+  ingress:
+    - from:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: monitoring
+      ports:
+        - port: 9090            # metrics port
+```
+
+## Cilium Policies (extended capabilities)
+
+```yaml
+# Cilium L7 policy — allow only GET /api/* (not POST/DELETE)
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: order-service-l7
+  namespace: production
+spec:
+  endpointSelector:
+    matchLabels:
+      app: order-service
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            app: frontend
+      toPorts:
+        - ports:
+            - port: "8080"
+              protocol: TCP
+          rules:
+            http:
+              - method: GET
+                path: /api/.*
+```
+
+## Debugging Blocked Traffic
+
+```bash
+# Cilium: observe dropped packets in real-time
+kubectl -n kube-system exec -it $(kubectl -n kube-system get pods -l k8s-app=cilium -o jsonpath='{.items[0].metadata.name}') \
+  -- cilium monitor --type drop
+
+# Hubble (if installed): flows between pods
+hubble observe --namespace my-app --verdict DROPPED
+
+# Calico: check policy hits
+kubectl exec -n kube-system <calico-node-pod> -- calicoctl get networkpolicy -n my-app
+
+# Test connectivity manually
+kubectl run test-pod --image=curlimages/curl -it --rm --restart=Never -- \
+  curl -v http://payment-service.payments.svc.cluster.local:8080/health
+```
+
+## Policy Design Checklist
+
+- [ ] Default deny-all applied to namespace
+- [ ] DNS egress allowed (port 53 UDP+TCP)
+- [ ] All required ingress/egress explicitly whitelisted
+- [ ] Ingress controller namespace allowed where applicable
+- [ ] Monitoring (Prometheus) scrape allowed
+- [ ] Cross-namespace refs use `namespaceSelector` with metadata label
+- [ ] Labels used in policies exist on actual pods

package/areas/devops/kubernetes/skills/pod-troubleshooting/SKILL.md

@@ -0,0 +1,129 @@
+---
+name: pod-troubleshooting
+type: skill
+description: Systematic diagnosis of pod failures — CrashLoopBackOff, OOMKilled, Pending, ImagePullBackOff, and service connectivity issues.
+related-rules:
+  - resource-governance.md
+  - workload-security.md
+allowed-tools: Read, Bash
+---
+
+# Skill: Pod Troubleshooting
+
+> **Expertise:** Systematic K8s failure diagnosis — from symptom to root cause in under 10 commands.
+
+## When to load
+
+When a pod is not Running, a service is unreachable, or a deployment is stuck.
+
+## Diagnostic Decision Tree
+
+```
+Pod not Running?
+├── Status: Pending
+│   ├── No nodes match → check node selectors, taints, resource requests
+│   └── PVC not bound → check StorageClass, PV availability
+├── Status: CrashLoopBackOff
+│   ├── Exit code 0 → process exited cleanly but K8s restarts it → check command
+│   ├── Exit code 1 → app error → check logs
+│   ├── Exit code 137 → OOMKilled → increase memory limit
+│   └── Exit code 143 → SIGTERM not handled → fix graceful shutdown
+├── Status: ImagePullBackOff
+│   ├── Image doesn't exist → check tag/digest
+│   └── Registry auth fails → check imagePullSecret
+└── Status: Error / Init:Error
+    └── Init container failed → check init container logs
+```
+
+## Command Cheatsheet
+
+```bash
+# 1. Overview — what's wrong
+kubectl get pods -n <ns> -o wide
+kubectl describe pod <pod> -n <ns>          # events section is the first place to look
+
+# 2. Logs
+kubectl logs <pod> -n <ns>                  # current container
+kubectl logs <pod> -n <ns> --previous       # last crashed container (CrashLoop)
+kubectl logs <pod> -n <ns> -c <container>   # specific container in multi-container pod
+
+# 3. Exec into running pod
+kubectl exec -it <pod> -n <ns> -- /bin/sh
+
+# 4. Resource pressure check
+kubectl top nodes
+kubectl top pods -n <ns>
+
+# 5. Events (cluster-wide, sorted)
+kubectl get events -n <ns> --sort-by='.lastTimestamp' | tail -20
+
+# 6. Debug ephemeral container (no exec needed — distroless images)
+kubectl debug -it <pod> -n <ns> --image=busybox:latest --target=<container>
+
+# 7. Node-level debug
+kubectl debug node/<node-name> -it --image=ubuntu
+```
+
+## CrashLoopBackOff Runbook
+
+```bash
+# Step 1: Get exit code
+kubectl describe pod <pod> -n <ns> | grep -A5 "Last State:"
+
+# Step 2: Get crash logs (may only appear in --previous)
+kubectl logs <pod> -n <ns> --previous --tail=100
+
+# Step 3: Check if OOMKilled
+kubectl describe pod <pod> -n <ns> | grep -i "OOMKilled\|Reason:"
+# If OOMKilled → increase memory limit or find memory leak
+
+# Step 4: Check security context (common in restricted namespaces)
+# Error: "permission denied" or "operation not permitted" → readOnlyRootFilesystem or dropped capabilities
+```
+
+## Pending Pod Runbook
+
+```bash
+# Check why pod can't be scheduled
+kubectl describe pod <pod> -n <ns> | grep -A20 "Events:"
+
+# Common causes:
+# "Insufficient cpu/memory" → check node capacity and pod requests
+kubectl describe nodes | grep -A5 "Allocated resources:"
+
+# "node(s) had taints that the pod didn't tolerate"
+kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
+
+# "0/3 nodes are available: 3 node(s) didn't match node affinity"
+# → check pod nodeSelector / affinity vs node labels
+kubectl get nodes --show-labels
+```
+
+## Service Connectivity Runbook
+
+```bash
+# Is the service selecting the right pods?
+kubectl get endpoints <svc> -n <ns>       # should show pod IPs; empty = selector mismatch
+
+# Test DNS resolution from within cluster
+kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup <svc>.<ns>.svc.cluster.local
+
+# Test HTTP connectivity
+kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- curl -v http://<svc>.<ns>.svc.cluster.local:<port>/health
+
+# Check NetworkPolicy blocking traffic
+# Install Hubble CLI (Cilium) or use:
+kubectl exec -n kube-system <cilium-pod> -- cilium monitor --from-pod <src-pod>
+```
+
+## OOMKilled Prevention
+
+```bash
+# Find actual peak memory usage via metrics
+kubectl top pods -n <ns> --sort-by=memory
+
+# Check Vertical Pod Autoscaler recommendation (if VPA installed)
+kubectl describe vpa <name> -n <ns> | grep -A10 "Recommendation:"
+
+# Rule of thumb: limit = 1.5× observed peak; request = 0.6× observed peak
+```