k8s-agent-skills 1.2.0

package/skills/cilium-gateway/SKILL.md

@@ -0,0 +1,283 @@
+---
+name: cilium-gateway
+description: Use when creating Gateway API resources for ingress, configuring TLS termination or passthrough, setting up HTTP-to-HTTPS redirect or traffic splitting, integrating oauth2-proxy or ExternalDNS with Cilium, or debugging Cilium Gateway controller issues including the Programmed=False cosmetic bug.
+---
+
+# Cilium Gateway
+
+Cilium v1.19.4 — Gateway API implementation for ingress traffic via per-node Envoy.
+
+## Overview
+
+Cilium implements Gateway API v1.4.1 using per-node Envoy proxies with eBPF TPROXY interception. Supports GatewayClass, Gateway, HTTPRoute, GRPCRoute, TLSRoute, and ReferenceGrant. Host network mode exposes listeners directly on node IPs without a LoadBalancer Service. TLS termination, traffic splitting, and header modification all handled in Envoy.
+
+## CRDs Used
+
+### Gateway API (standard)
+| CRD | Version | Purpose |
+|-----|---------|---------|
+| `GatewayClass` | `gateway.networking.k8s.io/v1` | Class reference (parametersRef → `CiliumGatewayClassConfig`) |
+| `Gateway` | `gateway.networking.k8s.io/v1` | Shared LB listener — hostname, TLS, ports |
+| `HTTPRoute` | `gateway.networking.k8s.io/v1` | HTTP route rules — matches, filters, backends |
+| `GRPCRoute` | `gateway.networking.k8s.io/v1` | gRPC route rules |
+| `TLSRoute` | `gateway.networking.k8s.io/v1alpha2` | TLS passthrough routing by SNI (experimental) |
+| `ReferenceGrant` | `gateway.networking.k8s.io/v1beta1` | Allow cross-namespace references (Secret, Service) |
+
+### Cilium-specific
+| CRD | Version | Purpose |
+|-----|---------|---------|
+| `CiliumGatewayClassConfig` | `cilium.io/v2alpha1` | Cilium-specific GatewayClass parameters (envoy config, LB type, etc.) |
+| `CiliumEnvoyConfig` | `cilium.io/v2` | Low-level Envoy config (used internally by Gateway controller) |
+
+## Architecture
+
+```
+Internet → LB IP → any node → eBPF TPROXY → per-node Envoy → identity "ingress" → backend pod
+```
+
+- Traffic arrives at any node, eBPF intercepts via TPROXY using `ingress` identity
+- Per-node Envoy (DaemonSet or cilium-agent embedded) handles L7
+- Two policy enforcement points: `world → ingress` and `ingress → backend`
+- Source IP preserved in `X-Forwarded-For` and `X-Envoy-External-Address` headers
+
+## Prerequisites
+
+```yaml
+# Helm values needed
+kubeProxyReplacement: true
+gatewayAPI:
+  enabled: true
+```
+
+Gateway API v1.4.1 CRDs must be pre-installed:
+```bash
+kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.4.1/config/crd/standard/gateway.networking.k8s.io_*.yaml
+```
+
+## Host Network Mode
+
+Cilium 1.16+ — expose Gateway directly on host network (no LoadBalancer Service). **Requires Envoy hostNetwork.**
+
+```yaml
+gatewayAPI:
+  enabled: true
+  hostNetwork:
+    enabled: true
+    nodes:
+      matchLabels:
+        role: infra
+envoy:
+  enabled: true
+  hostNetwork: true
+  securityContext:
+    capabilities:
+      keepCapNetBindService: true
+      envoy:
+        - NET_BIND_SERVICE
+```
+
+**Privileged ports (≤1023):** Add `NET_BIND_SERVICE` capability to Envoy.
+
+## Common Patterns
+
+### Gateway with TLS Termination
+```yaml
+apiVersion: gateway.networking.k8s.io/v1
+kind: Gateway
+metadata:
+  name: cilium-gateway
+  namespace: cilium-gateway
+  annotations:
+    external-dns.alpha.kubernetes.io/target: 79.76.124.104
+spec:
+  gatewayClassName: cilium
+  listeners:
+  - name: http
+    protocol: HTTP
+    port: 80
+    hostname: "*.kubexa.tech"
+    allowedRoutes:
+      namespaces:
+        from: All
+  - name: https
+    protocol: HTTPS
+    port: 443
+    hostname: "*.kubexa.tech"
+    tls:
+      mode: Terminate
+      certificateRefs:
+      - name: kubexa-tech-tls
+        kind: Secret
+  allowedRoutes:
+    namespaces:
+      from: All
+```
+
+### HTTP → HTTPS Redirect
+```yaml
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: http-to-https
+  namespace: cilium-gateway
+spec:
+  parentRefs:
+  - name: cilium-gateway
+    sectionName: http
+  hostnames:
+  - "*.kubexa.tech"
+  rules:
+  - filters:
+    - type: RequestRedirect
+      requestRedirect:
+        scheme: https
+        statusCode: 301
+```
+
+### HTTPRoute with Backend
+```yaml
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: app-route
+  namespace: myapp
+spec:
+  parentRefs:
+  - name: cilium-gateway
+    namespace: cilium-gateway
+    sectionName: https
+  hostnames:
+  - app.kubexa.tech
+  rules:
+  - matches:
+    - path:
+        type: PathPrefix
+        value: /
+    backendRefs:
+    - name: app-service
+      port: 8080
+```
+
+### Cross-Namespace Reference (ReferenceGrant)
+```yaml
+apiVersion: gateway.networking.k8s.io/v1beta1
+kind: ReferenceGrant
+metadata:
+  name: allow-httproutes
+  namespace: cilium-gateway
+spec:
+  from:
+  - group: gateway.networking.k8s.io
+    kind: HTTPRoute
+    namespace: kube-system
+  to:
+  - group: ""
+    kind: Secret
+    name: kubexa-tech-tls
+```
+Needed when HTTPRoute is in a different namespace than the Gateway or the Secret.
+
+### oauth2-proxy Integration
+```yaml
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: hubble-ui
+  namespace: kube-system
+spec:
+  parentRefs:
+  - name: cilium-gateway
+    namespace: cilium-gateway
+    sectionName: https
+  hostnames:
+  - hubble.kubexa.tech
+  rules:
+  - backendRefs:
+    - name: oauth2-proxy-hubble
+      port: 4180
+```
+
+### Traffic Splitting
+```yaml
+spec:
+  rules:
+  - backendRefs:
+    - name: app-v1
+      port: 80
+      weight: 90
+    - name: app-v2
+      port: 80
+      weight: 10
+```
+
+### Header Modification
+```yaml
+spec:
+  rules:
+  - filters:
+    - type: RequestHeaderModifier
+      requestHeaderModifier:
+        set:
+        - name: X-Custom-Header
+          value: my-value
+        add:
+        - name: X-Trace-Id
+          value: "abc123"
+```
+
+### LB IPAM Integration with Gateway Addresses
+```yaml
+apiVersion: gateway.networking.k8s.io/v1
+kind: Gateway
+metadata:
+  name: my-gateway
+spec:
+  addresses:
+  - type: IPAddress
+    value: 172.18.0.140
+  gatewayClassName: cilium
+  listeners:
+  - ...
+```
+Or via annotation (deprecated): `io.cilium/lb-ipam-ips: "172.18.0.141"`
+
+## Known Issues
+
+### Programmed=False (Cosmetic)
+Cilium Gateway may show `PROGRAMMED=False` status even though routes work fine.
+- Gateway status: "Address not ready yet" / Programmed: False
+- HTTPRoute status: Accepted: True, ResolvedRefs: True
+- Traffic flows despite Programmed=False
+- Do not treat this as a failure — routes are functional
+
+### TLS Passthrough Source IP
+When using TLS passthrough, backends see Envoy IP (node IP) as source, not the client IP. This is inherent to TCP proxy mode.
+
+## Troubleshooting
+
+```bash
+# Check gateway status
+kubectl get gateway -A
+kubectl describe gateway <name>
+
+# Check HTTPRoute
+kubectl describe httproute <name>
+
+# Check operator logs for Gateway API errors
+kubectl logs -n kube-system deployments/cilium-operator | grep gateway
+
+# Check Envoy config
+kubectl get ciliumenvoyconfigs -A
+
+# Verify Gateway API CRDs installed
+kubectl get crd | grep gateway.networking.k8s.io
+```
+
+### Common Errors
+| Symptom | Cause | Fix |
+|---------|-------|-----|
+| GatewayClass not found | Gateway API CRDs not installed | Install v1.4.1 CRDs |
+| Secret "X" not found | Missing ReferenceGrant for cross-namespace Secret | Add ReferenceGrant in Secret's namespace |
+| BackendNotFound | Service doesn't exist or wrong namespace | Check `backendRefs` names |
+| Programmed=False | Cosmetic bug (see above) | Verify HTTPRoute shows Accepted/ResolvedRefs |
+| HostNetwork port clash | Port already in use | Use unique ports per Gateway resource |

package/skills/cilium-network/SKILL.md

@@ -0,0 +1,243 @@
+---
+name: cilium-network
+description: Use when configuring Cilium CNI networking, writing network policies, debugging pod connectivity issues, setting up LB IPAM or L2 announcements for Service LoadBalancer IPs, enabling BGP route advertisement, configuring transparent encryption or Hubble observability, or managing Cilium security features (host firewall, Local Redirect Policy, CiliumCIDRGroup). Not for Gateway API (use cilium-gateway).
+---
+
+# Cilium Network
+
+Cilium v1.19.4 — eBPF-based CNI, networking, and security for Kubernetes.
+
+## Overview
+
+Cilium provides pod networking via eBPF, replacing kube-proxy with socket-level load balancing. Security policies enforce L3-L7 rules based on identity (not IP), surviving pod churn. Hubble delivers observability. LB IPAM + L2/BGP announcements expose Services externally without a cloud LB.
+
+## Cilium CRDs (Network Domain)
+
+| CRD | API | Purpose |
+|-----|-----|---------|
+| `CiliumNetworkPolicy` | `cilium.io/v2` | Namespaced L3-L7 network policy (identity, HTTP/gRPC/Kafka, DNS) |
+| `CiliumClusterwideNetworkPolicy` | `cilium.io/v2` | Cluster-scoped L3-L7 network policy |
+| `CiliumEndpoint` | `cilium.io/v2` | Per-pod status: identity, labels, policy enforcement state |
+| `CiliumEndpointSlice` | `cilium.io/v2` | Groups CiliumEndpoints for large-scale clusters |
+| `CiliumIdentity` | `cilium.io/v2` | Security identity (labels → numeric ID) |
+| `CiliumNode` | `cilium.io/v2` | Node-level Cilium config (allocated CIDRs, health endpoints) |
+| `CiliumCIDRGroup` | `cilium.io/v2alpha1` | Named group of CIDRs for policy `fromCIDRSet`/`toCIDRSet` |
+| `CiliumLoadBalancerIPPool` | `cilium.io/v2` | LB IPAM pool — allocate LoadBalancer IPs |
+| `CiliumL2AnnouncementPolicy` | `cilium.io/v2alpha1` | L2 ARP/NDP announcement of LoadBalancer IPs (beta) |
+| `CiliumLocalRedirectPolicy` | `cilium.io/v2` | Node-local traffic redirect (DNS cache, node-local proxy) |
+| `CiliumBGPClusterConfig` | `cilium.io/v2alpha1` | BGP cluster-level config (ASN, listen port, node selector) |
+| `CiliumBGPPeerConfig` | `cilium.io/v2alpha1` | BGP peer — transport, auth, timers, AFI/SAFI |
+| `CiliumBGPAdvertisement` | `cilium.io/v2alpha1` | Advertise pod CIDRs / service LB IPs via BGP |
+| `CiliumBGPNodeConfig` | `cilium.io/v2alpha1` | Per-node BGP status (read-only) |
+| `CiliumEnvoyConfig` | `cilium.io/v2` | Envoy proxy config (L7 policy enforcement) |
+
+## Key Concepts
+
+### Routing Modes
+- **Overlay (VXLAN/Geneve):** Encapsulation, works on any infra. Default.
+- **Native routing:** Uses host routing table. Needs KubeSpan/IPsec/WireGuard for cross-node encryption. Faster.
+- **kube-proxy replacement:** `kubeProxyReplacement=true` — eBPF replaces iptables for services. Required for many features.
+
+### Identity-Based Security
+Cilium assigns a security identity to every pod based on labels. Policies match on identity, not IP. This survives pod churn.
+
+### Policy Enforcement Points
+- L3/L4: eBPF datapath (fast path)
+- L7: Envoy proxy (HTTP/gRPC/Kafka inspection)
+- `reserved:host` identity for kubelet probes, `reserved:world` for external traffic
+
+## CRD Usage Patterns
+
+### Network Policies
+```yaml
+# Namespaced: allow ingress from app=frontend to app=backend on port 80
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: allow-frontend
+  namespace: default
+spec:
+  endpointSelector:
+    matchLabels:
+      app: backend
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        app: frontend
+    toPorts:
+    - ports:
+      - port: "80"
+        protocol: TCP
+
+# Cluster-wide: allow kubelet health probes from host
+apiVersion: cilium.io/v2
+kind: CiliumClusterwideNetworkPolicy
+metadata:
+  name: allow-kubelet-health-probes
+spec:
+  endpointSelector: {}
+  ingress:
+  - fromEntities:
+    - host
+    - remote-node
+```
+Key endpoints: `fromEndpoints`, `toEndpoints`, `fromEntities` (host/world/cluster/remote-node/all), `fromCIDR`, `toCIDR`, `toFQDNs` (DNS), `fromCIDRSet` with `cidrGroupRef`.
+
+### L7 HTTP Policy
+```yaml
+spec:
+  endpointSelector:
+    matchLabels:
+      app: my-api
+  ingress:
+  - toPorts:
+    - ports:
+      - port: "8080"
+      rules:
+        http:
+        - method: "GET"
+          path: "/public/.*"
+```
+
+### LB IPAM + L2 Announcements
+```yaml
+# 1. IP pool
+apiVersion: cilium.io/v2
+kind: CiliumLoadBalancerIPPool
+metadata:
+  name: prod-pool
+spec:
+  blocks:
+  - cidr: "10.0.10.0/24"
+  serviceSelector:
+    matchLabels:
+      lb-site: prod
+
+# 2. L2 announcement policy
+apiVersion: cilium.io/v2alpha1
+kind: CiliumL2AnnouncementPolicy
+metadata:
+  name: prod-policy
+spec:
+  loadBalancerIPs: true
+  serviceSelector:
+    matchLabels:
+      lb-site: prod
+  nodeSelector:
+    matchLabels:
+      role: infra
+  interfaces:
+  - ^eth[0-9]+
+```
+**Key:** L2 announcements require `loadBalancerIPs: true` or `externalIPs: true`. Without an explicit service selector, all services match. Without node selector, all nodes are candidates.
+
+### L2 Announcement Lease Tuning
+```yaml
+l2announcements:
+  leaseDuration: 15s    # time before failover
+  leaseRenewDeadline: 5s
+  leaseRetryPeriod: 2s
+```
+Client rate limit must be sized: `QPS = #services * (1 / leaseRenewDeadline)`.
+
+### CiliumCIDRGroup
+```yaml
+apiVersion: cilium.io/v2alpha1
+kind: CiliumCIDRGroup
+metadata:
+  name: vpn-cidrs
+spec:
+  externalCIDRs:
+  - "10.48.0.0/24"
+---
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: from-vpn
+spec:
+  endpointSelector: {}
+  ingress:
+  - fromCIDRSet:
+    - cidrGroupRef: vpn-cidrs
+```
+
+### BGP Control Plane
+Enable via `bgpControlPlane.enabled=true` in Helm.
+
+```yaml
+apiVersion: cilium.io/v2alpha1
+kind: CiliumBGPClusterConfig
+metadata:
+  name: cilium-bgp
+spec:
+  nodeSelector:
+    matchLabels:
+      bgp: enabled
+  bgpInstances:
+  - name: "65000"
+    localASN: 65000
+    peers:
+    - name: "65001"
+      peerAddress: "10.0.0.1"
+      peerASN: 65001
+      peerConfigRef:
+        name: cilium-peer
+---
+apiVersion: cilium.io/v2alpha1
+kind: CiliumBGPPeerConfig
+metadata:
+  name: cilium-peer
+spec:
+  transport:
+    peerPort: 179
+  timers:
+    holdTimeSeconds: 90
+    keepAliveTimeSeconds: 30
+  families:
+  - afi: ipv4
+    safi: unicast
+---
+apiVersion: cilium.io/v2alpha1
+kind: CiliumBGPAdvertisement
+metadata:
+  name: bgp-adverts
+spec:
+  advertisements:
+  - advertisementType: "PodCIDR"
+    selector:
+      matchLabels:
+        bgp: enabled
+  - advertisementType: "Service"
+    selector:
+      matchExpressions:
+      - {key: "color", operator: In, values: [blue]}
+    service:
+      addressType: LoadBalancerIP
+```
+
+### Encryption
+- **IPsec:** `encryption.type=ipsec` — tunnel or wireguard encryption mode
+- **WireGuard:** `encryption.type=wireguard` — per-tunnel, simpler key mgmt
+- **Transparent Encryption:** node-to-node encryption without app changes
+
+### Hubble Observability
+Enable with `hubble.enabled=true`, `hubble.relay.enabled=true`, `hubble.ui.enabled=true`.
+Metrics: `hubble.metrics.enabled: [dns, drop, tcp, flow, http, node:true]`
+
+### Host Firewall
+```yaml
+hostFirewall:
+  enabled: true
+```
+Protects host network namespace with CiliumClusterwideNetworkPolicy (use `nodeSelector` to target).
+
+## Common Mistakes
+
+- **Forgot `loadBalancerIPs: true` in L2 policy** → nothing announced
+- **No client rate limit sizing for L2 announcements** → lease renewal fails at scale
+- **CNP `endpointSelector` empty in wrong namespace** → policy applies to nothing
+- **Using `protocol: UDP` instead of `protocol: UDP` in port rules** — Cilium uses uppercase `TCP`/`UDP`/`SCTP`
+- **`externalTrafficPolicy: Local` with L2 announcements** → traffic drops on nodes without local pods. Use `Cluster` instead.
+- **BGP: peer IP unreachable from all nodes** — BGP runs in host network, ensure L2 connectivity to peer
+- **BGP `nodeSelector` across all CiliumBGP* CRDs must match** — a node must be selected by cluster config, peer config, AND advertisement
+- **Hubble metrics missing desired protocols** — list explicit metrics `[dns, drop, tcp, flow, http]`

package/skills/cnpg/SKILL.md

@@ -0,0 +1,130 @@
+---
+name: cnpg
+description: Use when working with CloudNativePG (CNPG) for PostgreSQL on Kubernetes — creating or troubleshooting Cluster, Backup, ScheduledBackup, Pooler, or ImageCatalog resources; defining bootstrap, storage, backup, or affinity settings.
+---
+
+# CloudNativePG (CNPG)
+
+## Overview
+
+CloudNativePG operator manages HA PostgreSQL clusters on Kubernetes. API: `postgresql.cnpg.io/v1`. Deployed via Helm chart (latest: 0.28.2 \u2192 operator 1.29.1).
+
+## CRD Reference
+
+| CRD | Purpose |
+|-----|---------|
+| `Cluster` | HA PostgreSQL cluster (core resource) |
+| `Backup` | On-demand backup |
+| `ScheduledBackup` | Cron-based backup schedule |
+| `Pooler` | PgBouncer connection pooler |
+| `Database` | Database-level management within cluster |
+| `Publication` / `Subscription` | Logical replication |
+| `ImageCatalog` / `ClusterImageCatalog` | Extension image management (v1.29+) |
+| `FailoverQuorum` | Quorum-based failover settings |
+
+## Cluster Spec (Key Fields)
+
+```yaml
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+spec:
+  instances: 1                    # Replicas (1 = standalone primary)
+  imageName: ghcr.io/cloudnative-pg/postgresql:18
+
+  storage:
+    size: 10Gi
+    storageClass: ceph-block      # Omit or null for cluster default
+    resizePVC: true               # Allow storage resize via Helm upgrade
+
+  walStorage:                     # Optional \u2014 separate WAL volume
+    size: 5Gi
+    storageClass: ceph-block
+
+  bootstrap:
+    initdb:
+      database: app               # Default DB name
+      owner: app                  # Default owner user
+      postInitSQL:                # Run after init
+        - CREATE EXTENSION vector;
+
+  affinity:
+    nodeSelector:
+      kubernetes.io/hostname: worker-proxmox
+    podAntiAffinity:
+      type: preferred             # required or preferred
+      topologyKey: kubernetes.io/hostname
+
+  postgresql:
+    parameters:
+      shared_preload_libraries: vector  # Extensions
+    pg_hba:                             # Custom pg_hba (v1.29+ use podSelectorRefs)
+
+  primaryUpdateStrategy: unsupervised   # unsupervised (default) or switchover
+  primaryUpdateMethod: restart          # restart or switchover
+
+  monitoring:
+    enablePodMonitor: true              # Generates Prometheus PodMonitor
+
+  backup:
+    volumeSnapshot:
+      snapshotClass: csi-ceph-block     # CSI snapshot class
+```
+
+## Bootstrap Methods
+
+| Method | Use case |
+|--------|----------|
+| `initdb` | Fresh cluster from scratch |
+| `pg_basebackup` | Clone from existing CNPG cluster |
+| `recovery` | PITR from Barman/volumeSnapshot/plugin backup |
+| `pg_basebackup` via `recovery` | Bootstrap replica from another cluster |
+
+## Backup Methods
+
+| Method | Status | Details |
+|--------|--------|---------|
+| `barmanObjectStore` | **Deprecated** (removed v1.30) | S3/GCS/Azure via Barman Cloud |
+| `volumeSnapshot` | Current | CSI snapshots (recommended for Ceph, EBS, etc.) |
+| `plugin` | Current | CNPG-I gRPC plugin for backup/WAL/recovery |
+
+## Affinity / Scheduling Patterns
+
+- **nodeSelector** — Pin to specific node (common for stateful workloads)
+- **podAntiAffinity** — Spread replicas across nodes (`required` for HA, `preferred` for soft)
+- **topologySpreadConstraints** — Distribute across zones
+- **tolerations** — For tainted nodes (GPU, infra-only)
+
+Set per-instance overrides via `instances` + `nodeSelector` to pin specific roles.
+
+## ImageCatalog (v1.29+)
+
+```yaml
+apiVersion: postgresql.cnpg.io/v1
+kind: ClusterImageCatalog
+metadata:
+  name: postgres-extensions
+spec:
+  images:
+    - image: ghcr.io/my-org/postgres:18-pgvector
+      major: 18
+```
+
+Then reference in Cluster: `spec.imageCatalogRef: postgres-extensions`
+
+## Common Mistakes
+
+- **BarmanObjectStore as default** \u2014 deprecated, use `volumeSnapshot` or `plugin`
+- **No WAL storage** \u2014 important for HA. Add `walStorage` block for performance
+- **`resizePVC: false`** \u2014 prevents storage expansion on Helm upgrade
+- **`primaryUpdateStrategy: supervised`** \u2014 requires manual approval for switchover; use `unsupervised` for automatic
+- **Same storageClass for all workloads** \u2014 ceph-block fine for most, but immich needs local-fast for vector extension performance
+- **Missing `INHERITED_ANNOTATIONS` / `INHERITED_LABELS`** \u2014 empty is fine; only set if you want annotations propagated to PVCs
+
+## Version Mapping
+
+| Helm Chart | Operator | Release Notes |
+|------------|----------|---------------|
+| 0.28.2 | 1.29.1 | Latest (CVE fixes) |
+| 0.28.0 | 1.29.0 | ImageCatalog, podSelectorRefs, CNPG-I plugins |
+
+Check `helm list -n <ns>` for deployed chart version; cross-reference with [cloudnative-pg.io/release-notes](https://cloudnative-pg.io/release-notes/).