@raishin/vanguard-frontier-agentic 1.4.0 → 1.6.0

package/skills/kubernetes/kubernetes-network-architecture-review/SKILL.md

@@ -0,0 +1,84 @@
+---
+name: kubernetes-network-architecture-review
+description: Use this skill for Kubernetes cluster network architecture review across the dataplane (CNI choice, kube-proxy mode, IPAM, MTU, encapsulation, dual-stack), service routing surface (Service types, EndpointSlices, internalTrafficPolicy/externalTrafficPolicy, topology-aware routing, Ingress, Gateway API), in-cluster DNS (CoreDNS, NodeLocal DNSCache, ndots), multi-cluster topology (ClusterMesh, Submariner, MCS-API design choices), and connectivity observability and troubleshooting. Trigger when the user asks how to choose or change a CNI, why pod-to-pod or pod-to-service traffic fails, whether to migrate from Ingress to Gateway API, why DNS latency is high, how to size Pod or Service CIDRs, or how to design multi-cluster networking. Does NOT review NetworkPolicy content (delegate to cilium-network-policy-review) or perform live mutations (delegate to kubernetes-live-network-policy-guard / kubernetes-live-mesh-policy-guard).
+allowed-tools: Read Grep Glob WebFetch
+metadata:
+  author: "github: Raishin"
+  version: "0.1.0"
+  updated: "2026-05-07"
+  category: networking
+---
+
+# Kubernetes Network Architecture Review
+
+## Purpose
+
+Review Kubernetes cluster networking *as a system* — the choices that shape every pod's reachability, latency, and blast radius before any policy is written. This skill is about **design correctness, sizing, and operational traps** in the dataplane, service routing, DNS, and multi-cluster surface. Policy correctness is delegated.
+
+## Scope boundary
+
+In scope:
+
+- CNI selection and dataplane mode (overlay vs native routing, eBPF vs iptables, kube-proxy replacement).
+- IPAM mode and Pod/Service CIDR sizing, including dual-stack and IPv6.
+- MTU and encapsulation overhead, jumbo frames, fragmentation traps.
+- Service surface: ClusterIP / NodePort / LoadBalancer / ExternalName / headless, EndpointSlices, `internalTrafficPolicy`, `externalTrafficPolicy`, topology-aware routing, `sessionAffinity`.
+- Ingress vs Gateway API: GatewayClass selection, role-oriented model (infrastructure / cluster operator / application), HTTPRoute / GRPCRoute / TLSRoute, ReferenceGrant, GAMMA mesh integration.
+- In-cluster DNS: CoreDNS Corefile, NodeLocal DNSCache architecture and risks, `ndots:5` tail-latency trap, ExternalDNS handoff.
+- Multi-cluster networking topology: Cilium ClusterMesh, Submariner, KEP-1645 MCS-API — pick on basis of identity, address-overlap, and policy semantics.
+- Connectivity observability: kube-proxy metrics, conntrack, Hubble flow shape, dropped packets, MTU mismatches.
+- Troubleshooting playbooks (read-only): pod-to-pod, pod-to-Service, pod-to-external, NodePort path, conntrack exhaustion.
+
+Out of scope — delegate:
+
+- NetworkPolicy / CiliumNetworkPolicy / CiliumClusterwideNetworkPolicy content review → `cilium-network-policy-review`.
+- Istio mesh policy / ambient L7 → `istio-ambient-mesh-review`.
+- Live mutations on policy objects → `kubernetes-live-network-policy-guard`, `kubernetes-live-mesh-policy-guard`.
+- Pod `securityContext`, capabilities, host networking → `kubernetes-pod-spec-review`.
+
+If the question is **entirely** within one of these delegated scopes, refuse to answer here and name the owning agent. Do not partial-answer and append a handoff line.
+
+## Lean operating rules
+
+- Prefer live cluster evidence (`kubectl get nodes,svc,endpointslices,gateway,gatewayclass,httproute,configmap -A`, `cilium status`, `cilium-dbg bpf`, `hubble observe`, `kube-proxy --metrics-port`, `conntrack -L`) when a Kubernetes MCP server, `kubectl`, and node-shell access are available; otherwise fall back to upstream documentation (kubernetes.io, gateway-api.sigs.k8s.io, docs.cilium.io, coredns.io) and sanitized YAML.
+- Separate confirmed facts from inference. If the CNI version, kube-proxy mode, IPAM mode, MTU, or DNS pod count was not queried, say so.
+- Treat **pod and service CIDR sizing** as architectural one-way doors — changing a Pod CIDR after the cluster is in use generally requires a cluster rebuild on most CNIs. Demand evidence of growth headroom.
+- Treat **kube-proxy mode swap** (iptables ↔ IPVS ↔ nftables, or to Cilium kube-proxy replacement) as a connectivity-affecting rollout — sessions on existing connections may break depending on mode and conntrack handling.
+- Treat **MTU mismatch between underlay and overlay** as a silent failure mode — TCP handshakes succeed, then large payloads stall. Always check whether encapsulation overhead (VXLAN 50B, Geneve 60B, WireGuard 60B with IPsec extra) was subtracted from node MTU.
+- Treat **`externalTrafficPolicy: Local`** as a correctness-sensitive choice — preserves source IP but breaks load balancing if the matching pod is not on the receiving node, and exposes pod placement to clients via uneven load.
+- Treat **`internalTrafficPolicy: Local`** the same way for in-cluster traffic — silent black-holing if no local endpoint exists.
+- Treat **`ndots:5` plus search-path expansion** as the default DNS tail-latency root cause. Five negative lookups before the absolute name on every external query, multiplied by every pod, is the dominant DNS load on most clusters.
+- Treat **NodeLocal DNSCache OOM** as a node-wide DNS outage — packet-filtering rules redirect to an unhealthy cache pod until restart. Memory limits and PDB are mandatory.
+- Treat **Ingress Controller annotations** as non-portable — migrating to Gateway API requires a route-by-route rewrite, not a global flip; plan the migration as a controller-by-controller cutover with overlap.
+- Treat **multi-cluster pod CIDR collisions** as the first failure for any cross-cluster scheme — reject any design that does not declare non-overlapping Pod CIDRs (or explicit NAT) before discussing identity or policy.
+- Treat **Cilium ClusterMesh kvstore replication lag** as a silent failure — remote ServiceImports serve stale endpoint maps with no error surfaced. Connections succeed but route to removed/replaced pods after a scale event. Compare endpoint revisions across peers (`cilium-dbg kvstore get`) when ClusterMesh is in scope.
+- Treat **conntrack table exhaustion** on busy nodes and **AWS NAT Gateway port exhaustion** (~55k connections per destination IP) as silent drops — packets disappear without errors at the application layer.
+- Treat **topology-aware routing skew** when zone labels are missing or unevenly populated as a silent traffic concentration — the Auto mode silently falls back to cluster-wide routing or drops endpoints.
+- Treat **any pod egress to `169.254.169.254` (AWS / Azure IMDS)** or **`metadata.google.internal` (GCP)** as a credential-theft vector. Recommend IRSA / Workload Identity / Pod Identity before discussing any egress allow rule. Surface unblocked metadata-service reachability as a HIGH severity finding rather than silently delegating to policy review.
+- **Do not invent CLI flags or commands.** Reference only `kubectl`, `cilium`, `cilium-dbg`, `hubble`, `calicoctl`, `subctl`, `ip`, `conntrack`, `iptables`, `ipvsadm`, `nft`, `coredns`. For anything outside this set, ask the user for the help text or a doc link rather than guess.
+- Refuse user-initiated mutation requests ("just apply this for me") and credential offers (kubeconfig, tokens, peer Secrets). Name the live-mutation delegate; do not proceed.
+- Keep the answer scoped, reversible, least-privilege, and explicit about blockers, unknowns, and which delegate skill owns the next step.
+
+## References
+
+Load these only when needed:
+
+- [Dataplane and CNI choice](references/dataplane-and-cni.md) — use when reviewing CNI selection, kube-proxy mode, IPAM, MTU and encapsulation, dual-stack and IPv6.
+- [Service and Gateway API routing](references/service-gateway-routing.md) — use when reviewing Service types, EndpointSlices, `internalTrafficPolicy` / `externalTrafficPolicy`, topology-aware routing, Ingress vs Gateway API, GatewayClass selection.
+- [DNS and service discovery](references/dns-and-discovery.md) — use when reviewing CoreDNS Corefile, NodeLocal DNSCache, `ndots:5`, autopath, ExternalDNS.
+- [Multi-cluster and egress topology](references/multi-cluster-and-egress.md) — use when reviewing ClusterMesh, Submariner, MCS-API, egress-from-cluster topology, cross-cluster service discovery.
+- [Connectivity troubleshooting playbook](references/troubleshooting-playbook.md) — use when diagnosing pod-to-pod, pod-to-service, pod-to-external failures, intermittent latency, MTU stalls, conntrack exhaustion.
+- [Evidence path and tooling](references/mcp-and-evidence.md) — use when choosing live cluster evidence, identifying CNI/kube-proxy/IPAM/DNS state, or switching to documentation mode.
+- [Official sources](references/official-sources.md) — use when you need the authoritative Kubernetes / Gateway API / Cilium / CoreDNS reference list.
+
+## Response minimum
+
+Return, at minimum:
+
+- the **scoped target** (CNI, Service, Gateway, DNS, multi-cluster topology, troubleshooting),
+- the **evidence level** — labeled **per finding**, not response-level only: `live evidence` / `documentation-based` / `sanitized user evidence` / `inference`. A response may legitimately mix levels; each finding must carry its own.
+- the **architectural posture findings** (with severity: high / medium / low),
+- the **safest next actions** — keep them reversible; for irreversible changes (CIDR resizing, kube-proxy swap), call out a tested cutover plan,
+- the **rollback or fallback path**,
+- the **delegate handoff** when the next step is policy content, mesh policy, live mutation, or pod-spec — name the skill or agent that owns it,
+- the **assumptions and blockers** that prevent stronger conclusions — if CNI version, kube-proxy mode, IPAM mode, node MTU, or DNS pod count were not confirmed by live evidence, each MUST appear here as an explicit open assumption. This field is not optional.

package/skills/kubernetes/kubernetes-network-architecture-review/metadata.json

@@ -0,0 +1,34 @@
+{
+  "id": "kubernetes-network-architecture-review",
+  "name": "Kubernetes Network Architecture Review",
+  "type": "skill",
+  "provider": "kubernetes",
+  "harnesses": [
+    "codex",
+    "claude-code",
+    "cursor",
+    "gemini",
+    "kiro",
+    "other"
+  ],
+  "summary": "Review Kubernetes cluster network architecture: CNI and dataplane selection, kube-proxy mode and replacement, IPAM and CIDR sizing, MTU and encapsulation, dual-stack and IPv6, Service surface (EndpointSlices, internalTrafficPolicy, externalTrafficPolicy, topology-aware routing), Ingress to Gateway API migration, CoreDNS and NodeLocal DNSCache, multi-cluster topology, and connectivity observability and troubleshooting. Excludes NetworkPolicy content review and live mutations — those are delegated to cilium-network-policy-review and the live-guard agents.",
+  "source_type": "original",
+  "official_docs": [
+    "https://kubernetes.io/docs/concepts/services-networking/",
+    "https://kubernetes.io/docs/reference/networking/virtual-ips/",
+    "https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/",
+    "https://kubernetes.io/docs/concepts/services-networking/service-traffic-policy/",
+    "https://kubernetes.io/docs/concepts/services-networking/topology-aware-routing/",
+    "https://kubernetes.io/docs/concepts/services-networking/dual-stack/",
+    "https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/",
+    "https://gateway-api.sigs.k8s.io/",
+    "https://docs.cilium.io/en/stable/network/concepts/",
+    "https://docs.cilium.io/en/stable/network/kube-proxy-replacement/",
+    "https://coredns.io/plugins/kubernetes/"
+  ],
+  "security_notes": "CNI and Pod CIDR are one-way architectural choices on most stacks — resizing requires cluster rebuild. kube-proxy mode swap can break in-flight connections. MTU mismatch between underlay and overlay is a silent payload-stall failure. externalTrafficPolicy: Local preserves source IP but black-holes traffic when no local endpoint exists. NodeLocal DNSCache OOM produces a node-wide DNS outage via stale packet-filter redirect. Multi-cluster pod CIDR collisions break any cross-cluster scheme regardless of policy correctness. ndots:5 plus search path is the dominant cluster DNS load on most installations.",
+  "last_verified": "2026-05-07",
+  "path": "skills/kubernetes/kubernetes-network-architecture-review",
+  "author": "github: Raishin",
+  "version": "0.1.0"
+}

package/skills/kubernetes/kubernetes-network-architecture-review/references/dataplane-and-cni.md

@@ -0,0 +1,89 @@
+# Dataplane and CNI
+
+## Step 1 — Identify the dataplane and CNI
+
+Capture, in order:
+
+1. **CNI plugin and version** — `kubectl -n kube-system get pods -l k8s-app=cilium -o name` (or `calico-node`, `flannel`, `aws-node`, `azure-cns`, `gke-cni`). Different CNIs have entirely different dataplanes (eBPF vs iptables vs vendor SDN), so every later question depends on this answer.
+2. **kube-proxy presence and mode** — `kubectl -n kube-system get ds kube-proxy` and the `--proxy-mode` flag in its ConfigMap (`kubectl -n kube-system get cm kube-proxy -o yaml`). On Cilium kube-proxy replacement, the DaemonSet does not exist or is disabled.
+3. **Routing mode** — encapsulation (VXLAN, Geneve, IPIP) vs native routing. Encapsulation works anywhere but adds 50–60 bytes of overhead and slightly higher CPU; native routing requires the underlay to route Pod CIDRs.
+4. **IPAM mode** — `cluster-pool` (CNI manages CIDR), `kubernetes` (uses `node.spec.podCIDR`), `aws-eni` / `azure` / `gke` (cloud IPAM, pods get VPC IPs).
+5. **Pod and Service CIDRs** — `kubectl cluster-info dump | grep -E '(cluster-cidr|service-cluster-ip-range)'` or check kube-controller-manager flags. Once set, these are very hard to change.
+6. **Node MTU and overlay MTU** — `ip link show | grep mtu`, then the CNI overlay interface (`cilium_vxlan`, `flannel.1`). The overlay MTU should be `node MTU − encapsulation overhead`.
+
+## Step 2 — Stress-test the CNI choice against actual requirements
+
+Common decision points and the trap each one hides:
+
+- **Cilium with kube-proxy replacement** — requires a kernel new enough for the eBPF features used (varies by version; verify with `cilium status` and the version's [system requirements page](https://docs.cilium.io/en/stable/operations/system-requirements/)). On older kernels, parts of KPR fall back to legacy paths and you lose the performance argument.
+- **Calico in BGP mode** — requires the underlay to accept BGP from every node, or a route reflector. In cloud, this often means a peering VM. In a managed cluster, this is usually impossible — the cluster ends up in IPIP encapsulation, defeating the BGP choice.
+- **AWS VPC CNI** — pods get VPC IPs, so subnet sizing and ENI limits per instance type bound pod density. An m5.large can hold ~30 pods because of ENI/IP limits, not memory. This is the dominant pod-density ceiling on EKS.
+- **Azure CNI (legacy)** — pre-allocates pod IPs from the subnet at node join, exhausting the subnet long before the pods exist.
+- **Azure CNI Overlay** — pods get a separate overlay CIDR; nodes still consume VNet IPs. Works at scale but add it as a deliberate IPAM choice, not the default.
+- **GKE alias IPs** — pod range is a secondary range on the VPC; sizing is fixed per cluster and resizing requires recreation.
+
+## Step 3 — Stress-test kube-proxy mode
+
+| Mode | Strengths | Real failures |
+|---|---|---|
+| `iptables` (default) | Simple, broadly tested | Rule count grows linearly with `Services × Endpoints`. On large clusters (10k+ Services) full rule resync becomes a multi-second event; new Service propagation latency rises. |
+| `ipvs` | Hash-based lookup, scales to many Services. Multiple LB algorithms. | Requires `ip_vs` kernel modules loaded on every node. Some session-affinity edge cases differ from iptables. Conntrack still drives source-IP preservation. |
+| `nftables` | Modern netfilter framework; better incremental update than iptables. | Newer; not all distros have stable nftables tooling. Still relatively young in production at scale. |
+| `kernelspace` (Windows) | Native Windows | Windows-only; behavior differs (no init container hostNetwork tricks, etc.). |
+| Cilium kube-proxy replacement | eBPF socket-LB; bypasses iptables entirely; preserves source IP for NodePort without `externalTrafficPolicy: Local` quirks. | Requires Cilium dataplane. Some hostPort and kernel-version edge cases. Verify `cilium status` reports KPR enabled in the expected mode (`Strict`, `Probe`, etc.). |
+
+Stress-tests the review must apply:
+
+- Migrating from `iptables` to `ipvs` or `nftables` on a running cluster — short connectivity blip during the rollout as kube-proxy rewrites rules. Schedule like a node-by-node rollout, not a config flag flip.
+- Migrating to Cilium KPR — uninstall kube-proxy *after* Cilium reports KPR healthy on every node. Removing kube-proxy first leaves Service VIPs unreachable.
+- Mixed mode during rollout — half nodes on iptables, half on Cilium KPR — Service traffic to a Pod on a KPR node from a kube-proxy node may follow different return paths and confuse conntrack. Plan a fast rollout window.
+
+## Step 4 — Stress-test IPAM and CIDR sizing
+
+A cluster sized 10× too small for its eventual workload count is the most common architectural debt — and it is hard to fix.
+
+Sizing stress-tests:
+
+- **Pod CIDR size** vs **max nodes × max pods/node** — a `/16` Pod CIDR with a `/24` per node gives 256 nodes max. A `/22` per node gives 64 nodes. Many CNIs allocate a fixed-size block per node, so the right number is `nodes × pods_per_node × headroom`.
+- **Service CIDR size** — Services tend to grow faster than people predict (every Helm chart adds a few). A `/16` is fine; a `/20` is risky in any cluster running a service mesh, since per-namespace mesh control planes add Services.
+- **CIDR collision with on-prem or peer VPC** — pods in `10.0.0.0/8` cannot route to an on-prem `10.x.x.x` system. RFC 1918 collision checks must precede every cluster build.
+- **`100.64.0.0/10` and `198.18.0.0/15`** — use carrier-grade NAT and benchmarking ranges if RFC 1918 is exhausted; cloud providers generally tolerate them.
+- **IPv6 / dual-stack** — single-stack v6 only is rarely supported by ecosystem tools (registries, observability). Dual-stack is the practical choice; ensure `--service-cluster-ip-range` and `--cluster-cidr` carry both families and that EndpointSlices report both.
+
+## Step 5 — Stress-test MTU and encapsulation
+
+MTU mismatch is a silent failure. The TCP three-way handshake passes (small packets), then the first large response stalls forever because Path MTU Discovery (PMTUD) ICMP is dropped by a firewall.
+
+The arithmetic the review must enforce:
+
+- VXLAN: 50 bytes (8 VXLAN + 8 UDP + 20 IPv4 + 14 Ethernet but 14 is shared) → overlay MTU = node MTU − 50.
+- Geneve: 60 bytes typical → overlay MTU = node MTU − 60.
+- IPIP: 20 bytes.
+- WireGuard (Cilium transparent encryption): ~60 bytes plus alignment.
+- IPsec: variable, ~73 bytes worst case.
+
+Stress-tests:
+
+- Cloud underlay at MTU 1500 → overlay should be 1450 (VXLAN). Some installers default to 1500 on the overlay too — silent corruption on first large packet.
+- Jumbo frames (MTU 9000) on AWS — works inside the same Availability Zone. Cross-AZ traffic is silently capped at 1500 by AWS, so jumbo overlay across AZs causes random stalls.
+- GKE / EKS / AKS managed nodes — verify the cloud-provider MTU before trusting the CNI's auto-detection. Some installers read `eth0` MTU at startup and miss later changes.
+
+Verification step: `kubectl run mtu-test --rm -it --image=alpine -- ping -M do -s <payload> <peer-pod-ip>`. The Don't-Fragment bit forces the kernel to fail rather than fragment, exposing the MTU ceiling.
+
+## Step 6 — Dual-stack and IPv6
+
+- The cluster API server, kube-controller-manager, kube-proxy, kubelet, and the CNI must all be configured for dual-stack. A partial enablement leads to confusing behavior where Services have only IPv4 addresses but Pods have both.
+- `Service.spec.ipFamilies` and `ipFamilyPolicy` (`SingleStack`, `PreferDualStack`, `RequireDualStack`) are the per-Service selectors. Default depends on the cluster's primary family.
+- EndpointSlices report addresses per family — verify both families are populated when `RequireDualStack` is in use.
+- IPv6-only clusters interact poorly with images pulled from IPv4-only registries; verify NAT64/DNS64 or registry mirroring is in place.
+
+## Output for this section
+
+- CNI and version,
+- kube-proxy mode (or KPR mode),
+- routing mode (encapsulation type or native),
+- IPAM mode and Pod / Service CIDR sizes,
+- node and overlay MTU,
+- dual-stack posture,
+- findings on sizing headroom, mode mismatch, MTU correctness, KPR readiness,
+- one-way-door warnings (CIDR resize, KPR migration) with cutover plan if a change is recommended.

package/skills/kubernetes/kubernetes-network-architecture-review/references/dns-and-discovery.md

@@ -0,0 +1,120 @@
+# DNS and Service Discovery
+
+## Step 1 — Identify the in-cluster DNS topology
+
+Capture:
+
+- CoreDNS deployment shape: `kubectl -n kube-system get deploy coredns -o wide`. Replica count, resources, anti-affinity.
+- CoreDNS Corefile: `kubectl -n kube-system get cm coredns -o yaml`. Plugins enabled, forward target, cache TTL, autopath, log/errors.
+- NodeLocal DNSCache presence: `kubectl -n kube-system get ds node-local-dns` (or `nodelocaldns`). Listening IP (typically a link-local address like `169.254.20.10`), upstream target, kube-proxy mode coupling.
+- kubelet `--cluster-dns` flag — does it point at the CoreDNS Service IP, or at the NodeLocal DNSCache link-local IP?
+- A pod's `/etc/resolv.conf`: `kubectl exec <pod> -- cat /etc/resolv.conf`. Note the `search`, `nameserver`, and `options ndots:` lines.
+
+## Step 2 — Stress-test the Corefile
+
+A canonical Corefile for a Kubernetes cluster typically uses these plugins in order:
+
+```
+.:53 {
+    errors
+    health { lameduck 5s }
+    ready
+    kubernetes cluster.local in-addr.arpa ip6.arpa {
+        pods insecure          # consider: pods verified
+        fallthrough in-addr.arpa ip6.arpa
+        ttl 30
+    }
+    prometheus :9153
+    forward . /etc/resolv.conf {
+        max_concurrent 1000
+    }
+    cache 30
+    loop
+    reload
+    loadbalance
+}
+```
+
+Stress-tests:
+
+- `pods insecure` returns A records for any pod IP without verifying a pod exists in that namespace. `pods verified` validates against pod existence in the same namespace; higher memory cost but a tighter security posture.
+- `forward . /etc/resolv.conf` follows the node's resolv.conf, which may point to the cloud DNS (169.254.169.254 on AWS, 168.63.129.16 on Azure). If pods talk to external services, every miss escapes the cluster — set `max_concurrent` deliberately to bound load on the upstream.
+- Missing `cache` plugin — every query hits upstream, including repeated queries for the same name within the TTL. The cache plugin is required for any cluster with non-trivial DNS load.
+- `cache` TTL larger than the upstream record's TTL — stale records persist past the authoritative source's update window. 30 seconds is a typical compromise.
+- `autopath` plugin — server-side search-path completion. Reduces the `ndots:5` round-trip cost (see Step 4) but requires `pods verified`, costs more memory, and complicates debugging because client-side lookups no longer match what reaches CoreDNS.
+- `loop` plugin — detects forwarding loops at startup. Without it, a forward to the cluster's own resolv.conf can loop until the deployment crashes. Always keep `loop` enabled.
+- `reload` plugin — picks up Corefile changes without a restart. Without it, a ConfigMap edit is not honored until the pod is recreated.
+
+## Step 3 — Stress-test CoreDNS scaling
+
+CoreDNS is the single most overlooked source of cluster-wide latency. Defaults from `kubeadm` and many installers are not production-sized.
+
+- Replica count — small clusters get 2 replicas by default. A cluster with 1000 pods doing 100 QPS per pod sees 100k QPS; two replicas with stock resources will become a queue.
+- Resource requests / limits — many installers default to a tight `100m CPU / 70Mi memory` request that is fine for 100 pods and a CPU throttle for 10000.
+- Pod anti-affinity — every CoreDNS replica must be on a different node. The default deployments usually have this; verify after migration to a new cluster.
+- PodDisruptionBudget — `minAvailable: 1` is the absolute floor; a stricter `maxUnavailable: 1` plus PDB on the DaemonSet's underlying nodes is safer during cluster autoscaler events.
+- `cluster-proportional-autoscaler` ([repository](https://github.com/kubernetes-sigs/cluster-proportional-autoscaler)) is the standard way to scale CoreDNS replicas with cluster size.
+
+## Step 4 — The `ndots:5` and search-path tail-latency trap
+
+The default `dnsPolicy: ClusterFirst` injects an `options ndots:5` line into every pod's `/etc/resolv.conf` along with a search list like:
+
+```
+search default.svc.cluster.local svc.cluster.local cluster.local
+options ndots:5
+```
+
+The `ndots:5` directive means: any name with fewer than 5 dots is treated as relative and tried against every search-list entry first.
+
+For a query like `api.example.com` (3 dots) the resolver issues:
+
+1. `api.example.com.default.svc.cluster.local.` — NXDOMAIN
+2. `api.example.com.svc.cluster.local.` — NXDOMAIN
+3. `api.example.com.cluster.local.` — NXDOMAIN
+4. `api.example.com.` — finally the real query
+
+For external services, that is **4× the DNS load** and 3× the chance of dropped UDP packets causing a 5-second timeout.
+
+Mitigations the review must consider:
+
+- Lower `options ndots:1` per-pod via `dnsConfig.options` for workloads that overwhelmingly resolve external hostnames. Cluster-internal Service names still resolve because they are absolute when written in the canonical `service.namespace.svc.cluster.local` form.
+- Use `ExternalName` Services so that `mysvc.namespace.svc.cluster.local` is the canonical reference, avoiding search-list expansion entirely.
+- Enable `autopath` in CoreDNS — completes the search path server-side, requires `pods verified` and more memory.
+- Always pair this fix with NodeLocal DNSCache so the search-list expansion cost stays on the node.
+
+## Step 5 — NodeLocal DNSCache deep-dive
+
+Per the [Kubernetes docs](https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/), NodeLocal DNSCache runs as a DaemonSet listening on a link-local IP. It solves three production problems:
+
+1. **conntrack overhead** — every UDP DNS query allocates a conntrack entry that lives 30 seconds; high-DNS-QPS pods exhaust conntrack tables and trigger random drops elsewhere. NodeLocal DNSCache upgrades to TCP for upstream traffic, where entries are removed on connection close.
+2. **5-second timeout amplification** — a dropped UDP packet triggers the resolver's retry timer (3 × 10s on glibc, with extra hops). Local cache hits remove most of this.
+3. **DNAT bypass** — pods talking to the link-local IP skip the iptables/IPVS Service VIP rewrite entirely.
+
+Operational risks the review must call out:
+
+- **OOMKill is a node-wide DNS outage.** When the local cache pod is killed, the iptables rules that redirected DNS traffic to the cache stay in place pointing at an unhealthy pod until the new pod is ready. Set memory limits with headroom (default cache is 10k entries ≈ 30 MB; 100 MB request and 200 MB limit is a safer baseline) and monitor `coredns_dns_cache_size_entries`.
+- **PodDisruptionBudget** for the DaemonSet — node drains during cluster autoscaler events should not all evict the local cache simultaneously.
+- **kubelet `--cluster-dns` flag must be updated** if the kube-proxy mode is IPVS — see the linked docs for the exact rewrite. iptables mode tolerates either the cache IP or the kube-dns Service IP.
+- **IPv6** — the link-local address must be enclosed in brackets (`[fd00::1]:53`).
+- **Cilium kube-proxy replacement** — Cilium's `socketLB` honors the per-pod redirect to the link-local IP only with the right options; verify with Cilium's NodeLocal DNSCache integration page for the version in use.
+
+## Step 6 — ExternalDNS and the boundary
+
+`ExternalDNS` runs in the cluster but operates on cloud DNS (Route 53, Cloud DNS, Azure DNS, OCI DNS) — it is the bridge between Kubernetes Service / Ingress / Gateway and externally reachable hostnames.
+
+Architecture review checks:
+
+- The IAM/role binding ExternalDNS uses must be scoped to the specific hosted zone(s) and the record types it actually creates (typically A, AAAA, CNAME, TXT). A `*` permission is over-scoped.
+- Ownership records (TXT) prevent two clusters fighting over the same hostname. Without them, two ExternalDNS deployments will continually overwrite each other's records — silent ping-pong.
+- TTL: the default may be high (300s); for blue/green or canary cutovers, a low TTL on the routed records is required, set per-resource via the `external-dns.alpha.kubernetes.io/ttl` annotation.
+
+This boundary is also where this skill's scope ends — DNS *outside* the cluster is the AWS / Azure / OCI network architect's territory.
+
+## Output for this section
+
+- CoreDNS topology, Corefile critical-plugin checks (cache, loop, reload, kubernetes plugin mode),
+- replica count and autoscaler posture,
+- `ndots:5` exposure for the workloads in scope and the per-pod `dnsConfig.options` plan,
+- NodeLocal DNSCache presence, memory headroom, PDB, and OOM exposure,
+- ExternalDNS scope, ownership-TXT posture, TTL hygiene,
+- findings, severity, and the next-step delegate (cloud-network-architect agent for hosted-zone scoping).

package/skills/kubernetes/kubernetes-network-architecture-review/references/mcp-and-evidence.md

@@ -0,0 +1,53 @@
+# Evidence Path and Tooling
+
+## Evidence path
+
+1. Prefer live cluster evidence when a Kubernetes MCP server, `kubectl`, the CNI's own CLI (`cilium`, `calicoctl`), Hubble, and (when in scope) node-shell access are available.
+2. Fall back to upstream documentation for canonical behavior: kubernetes.io for Service/EndpointSlice/Gateway concepts, gateway-api.sigs.k8s.io for resource semantics and stability channels, docs.cilium.io for dataplane and ClusterMesh, coredns.io for plugin behavior.
+3. Ask only for sanitized YAML (Service, EndpointSlice, Gateway, HTTPRoute, IngressClass, GatewayClass, CoreDNS ConfigMap, kube-proxy ConfigMap) when current-state proof matters and live access is absent.
+4. Label every conclusion as `live evidence`, `documentation-based`, `sanitized user evidence`, or `inference`.
+
+## Cluster state to confirm before the review
+
+- **Kubernetes version** — `kubectl version`. Networking semantics (EndpointSlice features, kube-proxy modes, Gateway API GA channel) gate on the version.
+- **CNI plugin and version** — `kubectl -n kube-system get pods -l k8s-app=cilium` (or `calico-node`, `aws-node`, etc.) and the agent's `version` subcommand. The dataplane is the rest of the conversation.
+- **kube-proxy mode (or replacement)** — `kubectl -n kube-system get cm kube-proxy -o yaml | grep -i mode`. On Cilium KPR, the DS may be absent.
+- **IPAM mode** — Cilium: `cilium config view | grep -iE 'ipam|cluster-pool|pod-cidr'`. Calico: `calicoctl get ippool`. Cloud-native CNIs: vendor docs.
+- **Pod and Service CIDRs** — `kubectl cluster-info dump | grep -E '(cluster-cidr|service-cluster-ip-range)'` or kube-controller-manager flags.
+- **Node MTU and overlay MTU** — `ip link show` on a node; `cilium config view | grep -i mtu`.
+- **Dual-stack posture** — `kubectl get svc -A -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.ipFamilies}{"\n"}{end}'`.
+- **CoreDNS replica count, Corefile, autoscaler** — `kubectl -n kube-system get deploy coredns`, `cm coredns`.
+- **NodeLocal DNSCache presence** — `kubectl -n kube-system get ds node-local-dns`.
+- **Gateway API CRDs and channel** — `kubectl get crd | grep gateway.networking.k8s.io` plus the controller version.
+- **Multi-cluster posture** — Cilium ClusterMesh: `cilium clustermesh status`. Submariner: `subctl show all`. Istio multi-cluster: control-plane endpoints.
+- **Hubble enabled** — required for cross-cluster flow observability.
+
+## Sanitization rules
+
+- Never request kubeconfig contents, ClusterMesh peer Secrets, Cilium agent tokens, or service account JWTs.
+- Replace cluster IDs, peer cluster URLs, public LB IPs, FQDNs, and namespace names with placeholders unless the user provides them in sanitized form.
+- Do not print Cilium or Istio control-plane Secrets, service account tokens, or registration manifests.
+- Do not exfiltrate `coredns` upstream resolver IPs from cloud metadata when those are sensitive.
+
+## Lightweight active probes (read-only) the review may suggest
+
+These probes touch only test pods or a debug DaemonSet. They mutate nothing operational.
+
+```shell
+# DNS resolution path on a pod, including search-list expansion
+kubectl run dnsprobe --rm -it --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- \
+  dig +short +trace <name>
+
+# Path MTU discovery from pod to peer
+kubectl run mtuprobe --rm -it --image=alpine -- \
+  ping -M do -c 3 -s <payload-bytes> <peer-ip>
+
+# Service VIP reachability and load distribution
+kubectl run vipprobe --rm -it --image=curlimages/curl -- \
+  sh -c 'for i in $(seq 50); do curl -s http://<svc>/ | grep -i pod-name; done | sort | uniq -c'
+
+# Hubble flow on a single pod over 60s
+hubble observe --from-pod <ns>/<pod> --since 60s
+```
+
+A review that recommends running any of these must explicitly mark them as ephemeral test pods, set `--rm`, and never recommend `--privileged` on a probe.

package/skills/kubernetes/kubernetes-network-architecture-review/references/multi-cluster-and-egress.md

@@ -0,0 +1,69 @@
+# Multi-Cluster Topology and Egress
+
+This reference is about *topology* — how clusters discover and address each other, and how pods leave the cluster — not about cross-cluster *policy*. Cross-cluster policy semantics (e.g. Cilium ClusterMesh `policy-default-local-cluster`) belong to `cilium-network-policy-review`.
+
+## Step 1 — Capture the multi-cluster posture
+
+- Number of clusters and their relationship: peers (mesh), hub-and-spoke, hierarchical.
+- Pod CIDRs per cluster and Service CIDRs per cluster — non-overlap is the prerequisite for every multi-cluster scheme.
+- Cross-cluster discovery mechanism in use: Cilium ClusterMesh, Submariner, Linkerd multi-cluster, Istio multi-primary or primary-remote, KEP-1645 Multi-Cluster Services API (MCS-API), or hand-rolled `ExternalName` chains.
+- Identity boundary: shared OIDC (workload identity federation), per-cluster identities with explicit trust, or no shared identity (in which case workload-identity review is required before cross-cluster authn is meaningful).
+- Underlay reachability: are pods directly routable across clusters (BGP, AWS VPC peering, VPN), or is traffic gateway-mediated (Submariner gateway, Istio east-west gateway)?
+
+## Step 2 — Stress-test pod / service CIDR overlap
+
+The first failure of every multi-cluster scheme is overlapping CIDRs. Reject any design that does not declare:
+
+- Per-cluster Pod CIDR ranges that do not intersect any peer's Pod CIDR.
+- Per-cluster Service CIDR ranges that do not intersect any peer's Service CIDR (relevant for ClusterMesh and MCS-API where a Service IP from cluster A may be routed from cluster B).
+- A reservation plan with documented headroom — adding a third cluster later usually requires rebuilding overlapping clusters.
+
+If overlap is unavoidable (e.g. existing brownfield clusters), the only correct answer is per-cluster NAT — Submariner's Globalnet and Istio's east-west gateway both implement variants of this, at the cost of losing client-IP visibility cross-cluster.
+
+## Step 3 — Stress-test the discovery mechanism choice
+
+| Mechanism | Strength | Real failure mode |
+|---|---|---|
+| Cilium ClusterMesh | eBPF-direct, no extra hops; identity propagation; Hubble flow visibility cross-cluster. | Requires Cilium on every cluster; CA trust setup; the `policy-default-local-cluster` flag flip is a documented landmine (see `cilium-network-policy-review`). **Silent-failure mode 1**: KVStoreMesh `--clustermesh-cache-ttl` defaults to `0s` which per upstream docs means "the cache is never revoked" — when connectivity to a remote cluster is lost, stale `ServiceImports` continue to serve removed endpoints indefinitely. Set a non-zero TTL explicitly. **Silent-failure mode 2**: `--global-ready-timeout` defaults to `10m` — clusters report ready even if remote sync has not converged. Verify with `cilium clustermesh status` and `cilium-dbg troubleshoot clustermesh` (direct mode) or `kubectl -n kube-system exec deploy/clustermesh-apiserver -c kvstoremesh -- clustermesh-apiserver kvstoremesh-dbg troubleshoot` (KVStoreMesh mode). |
+| Submariner | CNI-agnostic, can stitch heterogeneous clusters; Globalnet handles CIDR overlap. | Gateway nodes are choke points (single tunnel pair per gateway); MTU subtraction for the IPsec tunnel must be planned; service discovery via Lighthouse adds DNS hops. |
+| Istio multi-primary / primary-remote | Mesh-native; mTLS-everywhere; locality-aware load balancing. | Operationally complex; East-West Gateway is another L7 hop; cross-cluster health-checking depends on Istio version. |
+| Linkerd multi-cluster | Simple model (mirrored Services); no central control. | Mirror Service per cross-cluster Service grows the API server load; pod identity federation requires explicit trust setup. |
+| KEP-1645 MCS-API (`ServiceExport`, `ServiceImport`) | Vendor-neutral; standardized. | Implementations vary in maturity; check that the chosen controller (mcs-api-controller, AWS Cloud Map MCS, GKE Multi-Cluster Services) actually supports the channel needed. |
+| Hand-rolled `ExternalName` chains | No new infra. | No identity propagation; no health awareness; DNS rotation is the only failover; this is a smell, not a design. |
+
+## Step 4 — Stress-test egress topology
+
+How pods leave the cluster shapes IAM scoping, firewall rules, and observability.
+
+Patterns:
+
+- **Default node SNAT** — pods exit via the node's primary interface, source IP is the node IP. Simple, but every pod looks the same to external systems; firewall allowlists must allow the entire node range.
+- **CiliumEgressGatewayPolicy** — selected pods SNAT through a designated gateway node with a stable egress IP. Allows fine-grained external allowlisting. Stress-tests for this pattern belong to `cilium-network-policy-review`.
+- **AWS VPC CNI / Azure CNI native** — pods get VPC IPs and route directly to a NAT Gateway / NAT Router; egress IP is the NAT GW's. IAM must be scoped on a workload-identity basis since the source IP is shared.
+- **Per-namespace egress proxy** — an outbound HTTP/SOCKS proxy in each namespace; pods are forced through it via `HTTP_PROXY` env or NetworkPolicy. Adds a single audit point but is a single failure point.
+- **No egress** — air-gapped clusters where every external dependency is mirrored inside the cluster (registries, package mirrors, time servers). The only safe IAM posture for highly regulated workloads, but operationally heavy.
+
+Stress-tests the review must apply:
+
+- A pod with `HTTP_PROXY` set but `NO_PROXY` not configured to include `.svc.cluster.local` and the cluster CIDR — every in-cluster call goes through the egress proxy and back, doubling east-west latency.
+- An egress NAT Gateway shared across clusters in the same VPC — billing is per-GB on AWS; cross-AZ egress through a single AZ's NAT GW is a documented cost trap.
+- A managed cluster that disables NodeLocalDNS *and* uses VPC DNS — DNS queries become the dominant egress traffic; NAT GW cost spikes proportionally.
+- A workload that does TLS to an external service through the egress proxy without the proxy being a CONNECT-method TLS proxy — the proxy can only see SNI, not L7, and policy claims of "L7 control" are false.
+
+## Step 5 — East-west vs north-south
+
+Keep the language honest in any review:
+
+- **North-south** — traffic crossing the cluster boundary. Owners: Ingress / Gateway / LoadBalancer Service / egress topology.
+- **East-west** — traffic between pods in the same cluster (or cross-cluster but inside the multi-cluster trust domain). Owners: kube-proxy / Cilium dataplane / Service mesh.
+- **Diagonal** — a pod calling out to a managed cloud service (RDS, Cosmos DB, OCI Autonomous DB) or a SaaS API. This is north-south even though it does not feel like it; cloud network policy applies, not just NetworkPolicy.
+
+Many reviews conflate these. Naming the direction explicitly clarifies which controllers, which policy formats, and which observability pane apply.
+
+## Output for this section
+
+- Cluster relationship map and CIDR plan (no overlap),
+- discovery mechanism and identity boundary,
+- egress topology and source-IP plan,
+- findings on overlap, gateway choke points, IAM scoping for shared egress,
+- explicit handoffs: policy correctness → `cilium-network-policy-review` or `istio-ambient-mesh-review`; IAM scoping → workload-identity-review; cloud-side network architecture → AWS/Azure/OCI network architect agents.

package/skills/kubernetes/kubernetes-network-architecture-review/references/official-sources.md

@@ -0,0 +1,54 @@
+# Official Sources
+
+Load these only when needed.
+
+## Kubernetes upstream
+
+- [Services, Load Balancing, Networking concepts](https://kubernetes.io/docs/concepts/services-networking/) — the canonical entry point and topic taxonomy.
+- [Service](https://kubernetes.io/docs/concepts/services-networking/service/) — Service types, selectors, ports.
+- [EndpointSlices](https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/) — replaces the legacy Endpoints object; powers kube-proxy and Cilium socket-LB.
+- [Service Internal Traffic Policy](https://kubernetes.io/docs/concepts/services-networking/service-traffic-policy/) — `internalTrafficPolicy: Cluster | Local` semantics.
+- [Topology Aware Routing](https://kubernetes.io/docs/concepts/services-networking/topology-aware-routing/) — `service.kubernetes.io/topology-mode: Auto`.
+- [Dual-stack](https://kubernetes.io/docs/concepts/services-networking/dual-stack/) — `ipFamilies` and `ipFamilyPolicy`.
+- [Virtual IPs and Service Proxies (kube-proxy reference)](https://kubernetes.io/docs/reference/networking/virtual-ips/) — iptables, IPVS, nftables, kernelspace modes.
+- [DNS for Services and Pods](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/) — canonical DNS naming, search-list construction.
+- [NodeLocal DNSCache](https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/) — DaemonSet architecture, conntrack rationale, OOM risks.
+- [Pod Network](https://kubernetes.io/docs/concepts/cluster-administration/networking/) — cluster networking model.
+- [Network Plugins reference](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) — CNI integration.
+
+## Gateway API (sig-network)
+
+- [Gateway API home](https://gateway-api.sigs.k8s.io/) — resource taxonomy, role-oriented model, conformance.
+- [API stability and channels (Standard / Experimental)](https://gateway-api.sigs.k8s.io/concepts/versioning/) — what's GA vs experimental.
+- [GAMMA service mesh (Standard since v1.1.0)](https://gateway-api.sigs.k8s.io/mesh/gamma/) — routes attached directly to Services.
+- [Implementations](https://gateway-api.sigs.k8s.io/implementations/) — controller list and conformance reports.
+
+## Cilium
+
+- [Cilium documentation home](https://docs.cilium.io/en/stable/).
+- [Networking concepts (routing, IPAM, masquerading, MTU)](https://docs.cilium.io/en/stable/network/concepts/).
+- [Kube-proxy replacement](https://docs.cilium.io/en/stable/network/kube-proxy-replacement/) — eBPF socket-LB, requirements, mode (Strict / Probe), fallback paths.
+- [System requirements](https://docs.cilium.io/en/stable/operations/system-requirements/) — kernel feature matrix.
+- [ClusterMesh](https://docs.cilium.io/en/stable/network/clustermesh/) — multi-cluster service discovery and identity.
+- [Hubble Observability](https://docs.cilium.io/en/stable/observability/hubble/).
+
+## CoreDNS
+
+- [Plugin index (Corefile structure)](https://coredns.io/plugins/) — every plugin and its options.
+- [`kubernetes` plugin](https://coredns.io/plugins/kubernetes/) — `pods insecure | verified`, `endpoint_pod_names`, `ttl`, `fallthrough`, `autopath`.
+- [`forward` plugin](https://coredns.io/plugins/forward/) — `max_concurrent`, health-checking, policy.
+- [`cache` plugin](https://coredns.io/plugins/cache/) — TTL, success/denial/prefetch.
+
+## Multi-cluster
+
+- [KEP-1645: Multi-Cluster Services API](https://github.com/kubernetes/enhancements/blob/master/keps/sig-multicluster/1645-multi-cluster-services-api/README.md) — `ServiceExport`, `ServiceImport`.
+- [Submariner](https://submariner.io/) — CNI-agnostic multi-cluster gateway and Globalnet.
+- [Istio multi-cluster](https://istio.io/latest/docs/setup/install/multicluster/) — primary-remote, multi-primary topologies.
+
+## Cluster autoscaling for DNS
+
+- [`cluster-proportional-autoscaler`](https://github.com/kubernetes-sigs/cluster-proportional-autoscaler) — the canonical CoreDNS replica autoscaler.
+
+## Status of the Linux Foundation Kubernetes Network Engineer program
+
+As of 2026-05-07, the Linux Foundation's "[Kubernetes Network Engineer Program](https://training.linuxfoundation.org/kubernetes-network-engineer-program/)" page reports that the CNCF and Linux Foundation Education are *starting to work on* a Certified Kubernetes Network Engineer (CKNE) certification. **The published page does not yet list curriculum domains, learning objectives, or course modules.** This skill is therefore grounded in upstream Kubernetes / Gateway API / Cilium / CoreDNS documentation and not in the (unpublished) CKNE blueprint. When the LF publishes the CKNE domain list, the skill should be re-verified and the `last_verified` date bumped.