@raishin/vanguard-frontier-agentic 1.4.0 → 1.5.0

package/skills/kubernetes/kubernetes-network-architecture-review/references/multi-cluster-and-egress.md

@@ -0,0 +1,69 @@
+# Multi-Cluster Topology and Egress
+
+This reference is about *topology* — how clusters discover and address each other, and how pods leave the cluster — not about cross-cluster *policy*. Cross-cluster policy semantics (e.g. Cilium ClusterMesh `policy-default-local-cluster`) belong to `cilium-network-policy-review`.
+
+## Step 1 — Capture the multi-cluster posture
+
+- Number of clusters and their relationship: peers (mesh), hub-and-spoke, hierarchical.
+- Pod CIDRs per cluster and Service CIDRs per cluster — non-overlap is the prerequisite for every multi-cluster scheme.
+- Cross-cluster discovery mechanism in use: Cilium ClusterMesh, Submariner, Linkerd multi-cluster, Istio multi-primary or primary-remote, KEP-1645 Multi-Cluster Services API (MCS-API), or hand-rolled `ExternalName` chains.
+- Identity boundary: shared OIDC (workload identity federation), per-cluster identities with explicit trust, or no shared identity (in which case workload-identity review is required before cross-cluster authn is meaningful).
+- Underlay reachability: are pods directly routable across clusters (BGP, AWS VPC peering, VPN), or is traffic gateway-mediated (Submariner gateway, Istio east-west gateway)?
+
+## Step 2 — Stress-test pod / service CIDR overlap
+
+The first failure of every multi-cluster scheme is overlapping CIDRs. Reject any design that does not declare:
+
+- Per-cluster Pod CIDR ranges that do not intersect any peer's Pod CIDR.
+- Per-cluster Service CIDR ranges that do not intersect any peer's Service CIDR (relevant for ClusterMesh and MCS-API where a Service IP from cluster A may be routed from cluster B).
+- A reservation plan with documented headroom — adding a third cluster later usually requires rebuilding overlapping clusters.
+
+If overlap is unavoidable (e.g. existing brownfield clusters), the only correct answer is per-cluster NAT — Submariner's Globalnet and Istio's east-west gateway both implement variants of this, at the cost of losing client-IP visibility cross-cluster.
+
+## Step 3 — Stress-test the discovery mechanism choice
+
+| Mechanism | Strength | Real failure mode |
+|---|---|---|
+| Cilium ClusterMesh | eBPF-direct, no extra hops; identity propagation; Hubble flow visibility cross-cluster. | Requires Cilium on every cluster; CA trust setup; the `policy-default-local-cluster` flag flip is a documented landmine (see `cilium-network-policy-review`). **Silent-failure mode 1**: KVStoreMesh `--clustermesh-cache-ttl` defaults to `0s` which per upstream docs means "the cache is never revoked" — when connectivity to a remote cluster is lost, stale `ServiceImports` continue to serve removed endpoints indefinitely. Set a non-zero TTL explicitly. **Silent-failure mode 2**: `--global-ready-timeout` defaults to `10m` — clusters report ready even if remote sync has not converged. Verify with `cilium clustermesh status` and `cilium-dbg troubleshoot clustermesh` (direct mode) or `kubectl -n kube-system exec deploy/clustermesh-apiserver -c kvstoremesh -- clustermesh-apiserver kvstoremesh-dbg troubleshoot` (KVStoreMesh mode). |
+| Submariner | CNI-agnostic, can stitch heterogeneous clusters; Globalnet handles CIDR overlap. | Gateway nodes are choke points (single tunnel pair per gateway); MTU subtraction for the IPsec tunnel must be planned; service discovery via Lighthouse adds DNS hops. |
+| Istio multi-primary / primary-remote | Mesh-native; mTLS-everywhere; locality-aware load balancing. | Operationally complex; East-West Gateway is another L7 hop; cross-cluster health-checking depends on Istio version. |
+| Linkerd multi-cluster | Simple model (mirrored Services); no central control. | Mirror Service per cross-cluster Service grows the API server load; pod identity federation requires explicit trust setup. |
+| KEP-1645 MCS-API (`ServiceExport`, `ServiceImport`) | Vendor-neutral; standardized. | Implementations vary in maturity; check that the chosen controller (mcs-api-controller, AWS Cloud Map MCS, GKE Multi-Cluster Services) actually supports the channel needed. |
+| Hand-rolled `ExternalName` chains | No new infra. | No identity propagation; no health awareness; DNS rotation is the only failover; this is a smell, not a design. |
+
+## Step 4 — Stress-test egress topology
+
+How pods leave the cluster shapes IAM scoping, firewall rules, and observability.
+
+Patterns:
+
+- **Default node SNAT** — pods exit via the node's primary interface, source IP is the node IP. Simple, but every pod looks the same to external systems; firewall allowlists must allow the entire node range.
+- **CiliumEgressGatewayPolicy** — selected pods SNAT through a designated gateway node with a stable egress IP. Allows fine-grained external allowlisting. Stress-tests for this pattern belong to `cilium-network-policy-review`.
+- **AWS VPC CNI / Azure CNI native** — pods get VPC IPs and route directly to a NAT Gateway / NAT Router; egress IP is the NAT GW's. IAM must be scoped on a workload-identity basis since the source IP is shared.
+- **Per-namespace egress proxy** — an outbound HTTP/SOCKS proxy in each namespace; pods are forced through it via `HTTP_PROXY` env or NetworkPolicy. Adds a single audit point but is a single failure point.
+- **No egress** — air-gapped clusters where every external dependency is mirrored inside the cluster (registries, package mirrors, time servers). The only safe IAM posture for highly regulated workloads, but operationally heavy.
+
+Stress-tests the review must apply:
+
+- A pod with `HTTP_PROXY` set but `NO_PROXY` not configured to include `.svc.cluster.local` and the cluster CIDR — every in-cluster call goes through the egress proxy and back, doubling east-west latency.
+- An egress NAT Gateway shared across clusters in the same VPC — billing is per-GB on AWS; cross-AZ egress through a single AZ's NAT GW is a documented cost trap.
+- A managed cluster that disables NodeLocalDNS *and* uses VPC DNS — DNS queries become the dominant egress traffic; NAT GW cost spikes proportionally.
+- A workload that does TLS to an external service through the egress proxy without the proxy being a CONNECT-method TLS proxy — the proxy can only see SNI, not L7, and policy claims of "L7 control" are false.
+
+## Step 5 — East-west vs north-south
+
+Keep the language honest in any review:
+
+- **North-south** — traffic crossing the cluster boundary. Owners: Ingress / Gateway / LoadBalancer Service / egress topology.
+- **East-west** — traffic between pods in the same cluster (or cross-cluster but inside the multi-cluster trust domain). Owners: kube-proxy / Cilium dataplane / Service mesh.
+- **Diagonal** — a pod calling out to a managed cloud service (RDS, Cosmos DB, OCI Autonomous DB) or a SaaS API. This is north-south even though it does not feel like it; cloud network policy applies, not just NetworkPolicy.
+
+Many reviews conflate these. Naming the direction explicitly clarifies which controllers, which policy formats, and which observability pane apply.
+
+## Output for this section
+
+- Cluster relationship map and CIDR plan (no overlap),
+- discovery mechanism and identity boundary,
+- egress topology and source-IP plan,
+- findings on overlap, gateway choke points, IAM scoping for shared egress,
+- explicit handoffs: policy correctness → `cilium-network-policy-review` or `istio-ambient-mesh-review`; IAM scoping → workload-identity-review; cloud-side network architecture → AWS/Azure/OCI network architect agents.

package/skills/kubernetes/kubernetes-network-architecture-review/references/official-sources.md

@@ -0,0 +1,54 @@
+# Official Sources
+
+Load these only when needed.
+
+## Kubernetes upstream
+
+- [Services, Load Balancing, Networking concepts](https://kubernetes.io/docs/concepts/services-networking/) — the canonical entry point and topic taxonomy.
+- [Service](https://kubernetes.io/docs/concepts/services-networking/service/) — Service types, selectors, ports.
+- [EndpointSlices](https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/) — replaces the legacy Endpoints object; powers kube-proxy and Cilium socket-LB.
+- [Service Internal Traffic Policy](https://kubernetes.io/docs/concepts/services-networking/service-traffic-policy/) — `internalTrafficPolicy: Cluster | Local` semantics.
+- [Topology Aware Routing](https://kubernetes.io/docs/concepts/services-networking/topology-aware-routing/) — `service.kubernetes.io/topology-mode: Auto`.
+- [Dual-stack](https://kubernetes.io/docs/concepts/services-networking/dual-stack/) — `ipFamilies` and `ipFamilyPolicy`.
+- [Virtual IPs and Service Proxies (kube-proxy reference)](https://kubernetes.io/docs/reference/networking/virtual-ips/) — iptables, IPVS, nftables, kernelspace modes.
+- [DNS for Services and Pods](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/) — canonical DNS naming, search-list construction.
+- [NodeLocal DNSCache](https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/) — DaemonSet architecture, conntrack rationale, OOM risks.
+- [Pod Network](https://kubernetes.io/docs/concepts/cluster-administration/networking/) — cluster networking model.
+- [Network Plugins reference](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) — CNI integration.
+
+## Gateway API (sig-network)
+
+- [Gateway API home](https://gateway-api.sigs.k8s.io/) — resource taxonomy, role-oriented model, conformance.
+- [API stability and channels (Standard / Experimental)](https://gateway-api.sigs.k8s.io/concepts/versioning/) — what's GA vs experimental.
+- [GAMMA service mesh (Standard since v1.1.0)](https://gateway-api.sigs.k8s.io/mesh/gamma/) — routes attached directly to Services.
+- [Implementations](https://gateway-api.sigs.k8s.io/implementations/) — controller list and conformance reports.
+
+## Cilium
+
+- [Cilium documentation home](https://docs.cilium.io/en/stable/).
+- [Networking concepts (routing, IPAM, masquerading, MTU)](https://docs.cilium.io/en/stable/network/concepts/).
+- [Kube-proxy replacement](https://docs.cilium.io/en/stable/network/kube-proxy-replacement/) — eBPF socket-LB, requirements, mode (Strict / Probe), fallback paths.
+- [System requirements](https://docs.cilium.io/en/stable/operations/system-requirements/) — kernel feature matrix.
+- [ClusterMesh](https://docs.cilium.io/en/stable/network/clustermesh/) — multi-cluster service discovery and identity.
+- [Hubble Observability](https://docs.cilium.io/en/stable/observability/hubble/).
+
+## CoreDNS
+
+- [Plugin index (Corefile structure)](https://coredns.io/plugins/) — every plugin and its options.
+- [`kubernetes` plugin](https://coredns.io/plugins/kubernetes/) — `pods insecure | verified`, `endpoint_pod_names`, `ttl`, `fallthrough`, `autopath`.
+- [`forward` plugin](https://coredns.io/plugins/forward/) — `max_concurrent`, health-checking, policy.
+- [`cache` plugin](https://coredns.io/plugins/cache/) — TTL, success/denial/prefetch.
+
+## Multi-cluster
+
+- [KEP-1645: Multi-Cluster Services API](https://github.com/kubernetes/enhancements/blob/master/keps/sig-multicluster/1645-multi-cluster-services-api/README.md) — `ServiceExport`, `ServiceImport`.
+- [Submariner](https://submariner.io/) — CNI-agnostic multi-cluster gateway and Globalnet.
+- [Istio multi-cluster](https://istio.io/latest/docs/setup/install/multicluster/) — primary-remote, multi-primary topologies.
+
+## Cluster autoscaling for DNS
+
+- [`cluster-proportional-autoscaler`](https://github.com/kubernetes-sigs/cluster-proportional-autoscaler) — the canonical CoreDNS replica autoscaler.
+
+## Status of the Linux Foundation Kubernetes Network Engineer program
+
+As of 2026-05-07, the Linux Foundation's "[Kubernetes Network Engineer Program](https://training.linuxfoundation.org/kubernetes-network-engineer-program/)" page reports that the CNCF and Linux Foundation Education are *starting to work on* a Certified Kubernetes Network Engineer (CKNE) certification. **The published page does not yet list curriculum domains, learning objectives, or course modules.** This skill is therefore grounded in upstream Kubernetes / Gateway API / Cilium / CoreDNS documentation and not in the (unpublished) CKNE blueprint. When the LF publishes the CKNE domain list, the skill should be re-verified and the `last_verified` date bumped.

package/skills/kubernetes/kubernetes-network-architecture-review/references/service-gateway-routing.md

@@ -0,0 +1,108 @@
+# Service and Gateway API Routing
+
+## Step 1 — Identify the routing surface
+
+Capture:
+
+- `kubectl get services -A -o wide` — types, ClusterIPs, externalIPs, ports.
+- `kubectl get endpointslices -A` — confirm endpoints actually exist for every ClusterIP. A Service with zero EndpointSlices selects no pods.
+- `kubectl get ingress -A` and `kubectl get gateway,gatewayclass,httproute,grpcroute,tlsroute,referencegrant -A` — Ingress and Gateway API surface.
+- IngressClass and GatewayClass ownership — `kubectl get ingressclass`, `kubectl get gatewayclass` — which controllers actually exist on this cluster.
+
+## Step 2 — Stress-test Service type selection
+
+| Type | When right | When wrong |
+|---|---|---|
+| `ClusterIP` (default) | In-cluster only. | Used as a "fix" for east-west reachability when the real issue is NetworkPolicy or DNS. |
+| `NodePort` | Bare-metal or dev clusters where no cloud LB exists. | Used in cloud — exposes every node as an entry point, scraping the NodePort range from outside. Rarely the right cloud answer. |
+| `LoadBalancer` | Public/private endpoint via the cloud LB controller (or MetalLB / Cilium L2/BGP). | One LB per Service in classic mode is expensive at scale. Consolidate via Ingress/Gateway when feasible. |
+| `ExternalName` | DNS CNAME into an external host. | Used as policy-bypass — the pod still does its own DNS resolution; NetworkPolicy still applies to the resolved IP. |
+| Headless (`clusterIP: None`) | StatefulSet stable DNS, gRPC client-side LB, custom service discovery. | Used because someone wanted "no kube-proxy" — but then there is no load balancing, only DNS round-robin against EndpointSlices. |
+
+## Step 3 — Stress-test EndpointSlices
+
+EndpointSlices replace the legacy single-Endpoints object and are the data the Service VIP rules / eBPF maps actually consume.
+
+- A Service with `selector` populated but zero matching pods has an empty EndpointSlice — the VIP exists but every connection RSTs or hangs depending on kube-proxy mode. Common cause: Pod label drift after a Helm upgrade.
+- A Service without a `selector` (used for static external endpoints) requires a hand-maintained EndpointSlice; without it nothing is routed.
+- EndpointSlices report `Ready`, `Serving`, `Terminating` conditions. `externalTrafficPolicy: Local` plus rolling pod restart can transiently hit a node where the only local endpoint is `Terminating`. Verify ProxyTerminatingEndpoints behavior in the cluster's kube-proxy version.
+- Topology hints live in EndpointSlices (`hints.forZones`). They are populated by the EndpointSlice controller only when topology-aware routing prerequisites are met (multiple zones, sufficient endpoints per zone).
+
+## Step 4 — `internalTrafficPolicy` and `externalTrafficPolicy`
+
+These two fields are the single most-misunderstood part of Kubernetes networking.
+
+- **`externalTrafficPolicy: Cluster`** (default) — kube-proxy routes external traffic to any pod cluster-wide. Source IP is rewritten to the receiving node. Even load distribution.
+- **`externalTrafficPolicy: Local`** — kube-proxy only routes external traffic to pods on the same node as the receiving NodePort/LoadBalancer ingress point. Preserves source IP. **Black-holes traffic if no local endpoint exists** — health-check ports rely on this for cloud LB to drop the node from rotation. If health-check is misconfigured, the LB sends to a node with no pod, and the connection hangs.
+- **`internalTrafficPolicy: Cluster`** (default) — same as Cluster externally.
+- **`internalTrafficPolicy: Local`** — in-cluster traffic from a pod is routed only to a pod on the same node. **Silent black-hole** if no local endpoint exists; there is no LB rotation to fall back to. Used for node-local agent → app patterns where co-location is guaranteed by DaemonSet.
+
+Stress-tests the review must apply:
+
+- A Service with `externalTrafficPolicy: Local` and a Deployment of `replicas: 2` on a 10-node cluster — 8 nodes black-hole the NodePort. The cloud LB health-check must mark them unhealthy or every 8/10 connections fails.
+- `externalTrafficPolicy: Local` with `Local` health-check disabled on the LB — random failures.
+- `internalTrafficPolicy: Local` on a Service backing a Deployment with no anti-affinity — silent black-holes when no replica lands on the caller's node.
+
+## Step 5 — Topology-aware routing
+
+Three generations of API for keeping in-cluster traffic local:
+
+1. **`topologyKeys`** — first-generation Service field. **Removed** (not just deprecated) in Kubernetes 1.27. Clusters on 1.26 or earlier still using `topologyKeys` MUST plan migration before the 1.27 upgrade.
+2. **`service.kubernetes.io/topology-mode: Auto`** — annotation; replaced `topologyKeys`. Tells the EndpointSlice controller to populate `hints.forZones` so kube-proxy prefers same-zone endpoints, reducing cross-zone traffic cost. Per upstream kubernetes.io documentation, this annotation may itself be deprecated in favor of the next-generation `trafficDistribution` field.
+3. **`spec.trafficDistribution`** field on Service (KEP-4444) — newest API. If both `trafficDistribution` and `topology-mode: Auto` are set, the annotation overrides the field.
+
+Pick the API that matches your cluster's Kubernetes version; do not use `topologyKeys` even on 1.26 — the upgrade trap is too easy.
+
+Traps:
+
+- Auto mode requires sufficient endpoints per zone — too few endpoints, hints are not populated, no zone preference. Behavior silently changes when the deployment scales up.
+- A Service with topology hints behaves like `internalTrafficPolicy: Local` from a cost perspective but without the black-hole risk — same-zone preferred, fall back to cross-zone if local endpoints unhealthy.
+- Topology hints do not respect `externalTrafficPolicy: Local` — they are independent fields.
+
+## Step 6 — Ingress vs Gateway API
+
+Gateway API is the successor to Ingress. The migration is per-route, not per-cluster.
+
+What's GA today (per the [Gateway API site](https://gateway-api.sigs.k8s.io/) and the [versioning page](https://gateway-api.sigs.k8s.io/concepts/versioning/)):
+
+- `Gateway`, `GatewayClass`, `HTTPRoute` — Standard channel, graduated to `v1` in the v1.0.0 release.
+- `GRPCRoute` — **Standard channel since v1.1.0** (`v1`). Earlier sources that label it experimental refer to the v1.0 line.
+- `ReferenceGrant` — Standard channel (`v1beta1`).
+- GAMMA service-mesh integration (routes attach directly to a Service rather than a Gateway) — Standard channel since v1.1.0.
+
+Experimental channel as of the v1.x line — verify against the controller's conformance for the version in use:
+
+- `TLSRoute`, `TCPRoute`, `UDPRoute`, `BackendTLSPolicy`, `BackendTrafficPolicy`.
+
+Always confirm by listing the CRDs the controller actually installed: `kubectl get crd -l gateway.networking.k8s.io/bundle-version`.
+
+The role-oriented model:
+
+- **Infrastructure provider** owns `GatewayClass`.
+- **Cluster operator** owns `Gateway` instances (and the policies that bound them).
+- **Application developer** owns `HTTPRoute` / `GRPCRoute` / `TLSRoute` and attaches them to a Gateway via `parentRefs`.
+
+Migration stress-tests:
+
+- Ingress Controller annotations (`nginx.ingress.kubernetes.io/...`, `cert-manager.io/cluster-issuer`, `external-dns.alpha.kubernetes.io/...`) are non-portable. Migrating an Ingress to an HTTPRoute requires translating each annotation into either a built-in HTTPRoute field, a `BackendTLSPolicy`, or a controller-specific extension.
+- Cross-namespace references in HTTPRoute (`backendRefs` to a Service in a different namespace) require an explicit `ReferenceGrant` in the target namespace — without it, the reference is denied. This is a deliberate security posture; do not work around it with cluster-scoped routes.
+- Two GatewayClasses on the same cluster (e.g. `cilium` and `nginx`) — routes must select via `parentRefs.name`/`namespace`. Avoid having multiple Gateways listen on the same hostname.
+- TLS termination at the Gateway with `backendRefs` over plain HTTP — re-encryption to the backend requires `BackendTLSPolicy` (Experimental) or a service mesh; without it, in-cluster traffic is plaintext.
+- GAMMA mesh — HTTPRoute attached to a Service (not a Gateway) reshapes east-west traffic. Available implementations differ on which fields they honor; verify against the controller's GAMMA conformance report.
+
+## Step 7 — Session affinity, sourceIPHash, and gRPC
+
+- `Service.spec.sessionAffinity: ClientIP` plus `sessionAffinityConfig.clientIP.timeoutSeconds` keeps a client IP pinned to one endpoint via conntrack/IPVS hash. Useful for stateful protocols, hostile to deployment rollover (sessions stick to the old pod until timeout).
+- gRPC over a single HTTP/2 connection multiplexes RPCs over one TCP connection; a single Service VIP without client-side LB sticks every RPC to one backend. Headless Service plus client-side resolver (gRPC-Go / xDS) is the canonical answer; an L7 Gateway (gRPC-aware) is the other.
+- Long-lived WebSocket connections plus rolling deployment plus session affinity — every old-pod connection survives until the client disconnects. Plan termination grace and client-reconnect strategy.
+
+## Output for this section
+
+- Service surface inventory and types,
+- EndpointSlice health (selectors actually match pods),
+- `internalTrafficPolicy` and `externalTrafficPolicy` correctness against pod placement and replica count,
+- topology-aware routing readiness,
+- Ingress vs Gateway API posture, GA-channel resources in use vs experimental,
+- migration path with annotation-to-route mapping for the controller in scope,
+- session affinity and gRPC routing posture,
+- findings, severity, and the next-step delegate (e.g. NetworkPolicy review if a route depends on cross-namespace traffic).

package/skills/kubernetes/kubernetes-network-architecture-review/references/troubleshooting-playbook.md

@@ -0,0 +1,100 @@
+# Connectivity Troubleshooting Playbook (read-only)
+
+This playbook is for diagnosis. It does not mutate cluster state. Every command listed here either reads from the API server, reads from a node's kernel, or emits a controlled probe (a single ping, a single curl).
+
+## Symptom 1 — Pod-to-pod connectivity fails
+
+Hypothesis ladder, in order of cost:
+
+1. **DNS** — does the source pod resolve the destination's Service or pod-IP DNS name? `kubectl exec <src> -- nslookup <name>`. If this fails, jump to the DNS playbook below.
+2. **Service VIP exists and has endpoints** — `kubectl get svc <svc>` plus `kubectl get endpointslice -l kubernetes.io/service-name=<svc>`. Empty EndpointSlice → label selector mismatch.
+3. **NetworkPolicy** — `kubectl get networkpolicies,ciliumnetworkpolicies,ciliumclusterwidenetworkpolicies -A -o yaml`. If any policy selects either pod, default-deny may be active. **At this point, hand off to `cilium-network-policy-review`** — that skill owns policy correctness.
+4. **Underlying L3 reachability** — from the source node, `ip route get <dst-pod-ip>`; on Cilium, `cilium-dbg bpf endpoint list` and `cilium-dbg policy trace --src-pod ns/pod --dst-pod ns/pod`.
+5. **MTU** — `kubectl exec <src> -- ping -M do -s 1450 <dst-pod-ip>` then increment payload until packets are dropped. The largest size that succeeds is the path MTU.
+6. **Conntrack table full** — on the source node, `conntrack -L | wc -l` vs `sysctl net.netfilter.nf_conntrack_max`. Near 90% means new connections will be dropped.
+
+## Symptom 2 — Pod can reach Service VIP but only some endpoints
+
+Likely causes:
+
+- **`externalTrafficPolicy: Local` or `internalTrafficPolicy: Local`** — `kubectl get svc <svc> -o yaml | grep -i trafficpolicy`. With `Local`, pods on nodes without a local endpoint silently fail.
+- **EndpointSlice marked `Ready: false` or `Terminating: true`** — `kubectl get endpointslice -l kubernetes.io/service-name=<svc> -o yaml`. ProxyTerminatingEndpoints behavior depends on the kube-proxy / Cilium version.
+- **kube-proxy stale rules** — `kubectl -n kube-system logs <kube-proxy-pod>` for sync errors. On iptables mode, full resync runs every `syncPeriod`; a stuck sync leaves stale rules pointing at deleted pods.
+- **Topology hints with insufficient endpoints** — Auto mode does not populate hints when one zone has too few; behavior changes silently with deployment scale.
+
+## Symptom 3 — Pod-to-external service intermittent failure
+
+Hypothesis ladder:
+
+1. **DNS** — 5-second timeouts indicate UDP packet drops; check NodeLocal DNSCache health and conntrack pressure on the node. CoreDNS request latency: `coredns_dns_request_duration_seconds`.
+2. **Cloud metadata service** — `169.254.169.254` (AWS / Azure IMDS) or `metadata.google.internal` (GCP). **Flag as HIGH severity if any pod can reach it without an explicit NetworkPolicy deny.** Unblocked access lets any pod obtain the node's instance IAM credentials — credential-theft CVE class. Recommend IRSA (AWS), Workload Identity (GCP/Azure), or Pod Identity as the *first* remediation; the policy deny rule is delegated to `cilium-network-policy-review`, but the architectural posture finding must surface here, not be silently delegated.
+3. **NAT Gateway port exhaustion** — AWS NAT GW supports ~55k simultaneous connections per destination IP/port. Pods making many short connections to the same external endpoint can exhaust ports; symptom is intermittent SYN drops. Solution: Network Load Balancer instead of NAT GW for that destination, or VPC endpoint.
+4. **Egress firewall / WAF** — if a per-namespace egress proxy exists, check its logs.
+5. **Path MTU** — see Symptom 1, step 5; path-MTU issues to external endpoints often blame TLS but the failure is L3.
+
+## Symptom 4 — DNS-specific high latency
+
+Hypothesis ladder:
+
+1. **CoreDNS replicas saturated** — CPU throttling: `kube_pod_container_status_throttled_seconds_total{pod=~"coredns.*"}`. Insufficient replicas → autoscaler or manual bump.
+2. **`ndots:5` amplification** — `kubectl exec <pod> -- cat /etc/resolv.conf` to confirm `ndots`. External hostnames generate 4× lookups before the absolute name. Mitigations are in the DNS-and-discovery reference.
+3. **NodeLocal DNSCache OOM or absent** — `kubectl -n kube-system get pods -l k8s-app=node-local-dns -o wide` (or whatever the label is). If absent, every UDP packet adds conntrack pressure.
+4. **Upstream DNS slow** — the cluster's `forward .` target. CoreDNS `coredns_forward_request_duration_seconds` segments by upstream.
+5. **conntrack table full** — UDP DNS queries fill it.
+
+## Symptom 5 — Intermittent stalls on large payloads only
+
+Almost always MTU. The TCP handshake (small packets) succeeds, then the first response above the path MTU is dropped because Path MTU Discovery ICMP is filtered.
+
+Diagnose:
+
+- `kubectl exec <pod> -- ping -M do -s <bytes> <peer>` — find the MTU ceiling.
+- Check overlay MTU vs node MTU vs underlay. Encapsulation overhead must be subtracted from node MTU when configuring the overlay device (`cilium_vxlan`, `flannel.1`, `cni0`).
+- Cross-AZ on AWS — jumbo frames silently capped at 1500.
+
+Fix: configure the CNI overlay MTU correctly and restart the agent DaemonSet (rolling). On Cilium, `cilium config view | grep -i mtu` and Helm value `MTU` (the Cilium agent reads MTU from the host but explicit config is safer).
+
+## Symptom 6 — NodePort or LoadBalancer connection refused intermittently
+
+- `externalTrafficPolicy: Local` — health-check on the LB must be on the right port, otherwise the LB sends to nodes with no local endpoint.
+- IPVS mode and `sessionAffinity: ClientIP` — sessions persist past pod deletion until timeout.
+- Cilium kube-proxy replacement transition — verify `cilium status` reports KPR `Strict` (or expected mode) on every node.
+
+## Diagnostic commands (read-only, by component)
+
+```shell
+# API view
+kubectl get nodes -o wide
+kubectl -n kube-system get pods -o wide
+kubectl get svc,endpointslices,ingress,gateway,gatewayclass,httproute -A
+
+# CoreDNS
+kubectl -n kube-system get cm coredns -o yaml
+kubectl -n kube-system logs -l k8s-app=kube-dns --tail=200
+kubectl exec <pod> -- cat /etc/resolv.conf
+
+# kube-proxy
+kubectl -n kube-system get cm kube-proxy -o yaml
+kubectl -n kube-system logs -l k8s-app=kube-proxy --tail=200
+
+# Cilium
+kubectl -n kube-system exec ds/cilium -- cilium status --verbose
+kubectl -n kube-system exec ds/cilium -- cilium-dbg bpf endpoint list
+kubectl -n kube-system exec ds/cilium -- cilium-dbg policy trace --src-pod <ns>/<pod> --dst-pod <ns>/<pod>
+hubble observe --from-pod <ns>/<pod> --to-pod <ns>/<pod> --last 200
+
+# Node-level (use kubectl debug ephemeral container with --profile=netadmin — do NOT use --privileged)
+# Example: kubectl debug node/<node-name> -it --image=registry.k8s.io/e2e-test-images/agnhost:2.45 --profile=netadmin
+ip route
+ip link show
+conntrack -L | wc -l
+sysctl net.netfilter.nf_conntrack_max
+```
+
+## When the playbook hits the scope boundary
+
+- Policy correctness — hand off to `cilium-network-policy-review`.
+- Mesh L7 problems — hand off to `istio-ambient-mesh-review`.
+- Live mutation of policy — hand off to `kubernetes-live-network-policy-guard` or `kubernetes-live-mesh-policy-guard`.
+- Pod-spec hostNetwork / capabilities issues — hand off to `kubernetes-pod-spec-review`.
+- Cloud-side network problems (NAT GW, VPC peering, hosted DNS) — hand off to the cloud-provider network architect agent.

package/skills/velero/velero-backup-restore-guard/SKILL.md

@@ -20,7 +20,7 @@ Guard live Velero operations — restore execution, schedule deletion, BackupSto
 - Confirm cluster context (`kubectl config current-context`) and target namespace before any Velero operation — ambiguous context is a hard stop.
 - Capture current state of the target Backup, Schedule, or BSL (`velero backup describe <name> --details`, `kubectl get schedule <name> -o yaml`) before every write — Velero has no built-in undo.
 - For restore operations: require `includedNamespaces` to be explicitly scoped; a cluster-wide restore (`includedNamespaces: []`) requires explicit platform-team sign-off.
-- Recommend `velero restore create --dry-run` before every non-emergency restore; treat missing dry-run as a hard stop in non-emergency scenarios.
+- Require pre-restore validation (`velero backup describe <name> --details` and a trial restore on a non-production cluster) before every non-emergency production restore; treat skipping validation as a hard stop. **Velero has no `--dry-run` flag on `velero restore create`** — do not recommend one.
 - Block deleting a Schedule that is the only backup for a production namespace unless an alternative backup source is confirmed.
 - Block changing a BSL `default: true` without confirming no in-progress backups and reviewing the impact on all dependent Schedules.
 - Check pre-backup hook coverage on stateful workloads (PostgreSQL, MySQL, Kafka) — missing quiesce hooks mean inconsistent backups.
@@ -39,6 +39,6 @@ Load these only when needed:
 - Current state of the Backup/Schedule/BSL (evidence level)
 - Hard-stop assessment (is this a blocked operation?)
 - Explicit platform-team sign-off status
-- Recommended dry-run or safe-path command
+- Recommended validation step or safe-path command
 - Rollback posture
 - Post-operation verification steps

package/skills/velero/velero-backup-restore-guard/references/safety-checklist.md

@@ -8,7 +8,7 @@ Before executing any `velero restore create` command, confirm all 10 items. A si
 - [ ] **2. Namespace scope is explicit** — `includedNamespaces` lists one or more specific namespaces. Empty list (`[]`) = cluster-wide restore = requires explicit platform-team sign-off with ticket reference.
 - [ ] **3. Backup timestamp verified** — the backup name and creation timestamp have been confirmed as the correct recovery point. Do not restore from an older backup if a closer-in-time backup exists and is healthy.
 - [ ] **4. Backup phase is Completed** — `velero backup describe <name>` shows `Phase: Completed`. Do not restore from a `PartiallyFailed` or `Failed` backup without explicit acknowledgment of the incomplete scope.
-- [ ] **5. Dry-run executed and reviewed** — `velero restore create --dry-run` output has been reviewed for unexpected resource counts, namespace scope, and PV restore entries. (Exception: active P0 incident with explicit platform-team override.)
+- [ ] **5. Pre-restore validation executed** — `velero backup describe <name> --details` has been reviewed for resource counts, namespace scope, and PV entries; for material restores, a trial restore was first executed on a non-production cluster and inspected with `velero restore describe <name> --details` and `velero restore logs <name>`. **Velero has no `--dry-run` flag on `velero restore create`** — never substitute it. (Exception: active P0 incident with explicit platform-team override and ticket reference.)
 - [ ] **6. existingResourcePolicy reviewed** — if `existingResourcePolicy: update` is used, the approver understands this will overwrite live Secrets, ConfigMaps, RBAC objects, and ServiceAccounts in the target namespace.
 - [ ] **7. PV restore posture confirmed** — `restorePVs: true/false` intent is explicit. If false, stateful applications will start with empty persistent volumes.
 - [ ] **8. Current state captured** — target namespace resources have been exported (`kubectl get all,cm,secret,pvc -n <ns> -o yaml > pre-restore-state.yaml`) as a rollback artifact.

package/skills/velero/velero-backup-restore-guard/references/workflow-and-output.md

@@ -75,19 +75,28 @@ spec:
   restorePVs: true
 ```
 
-### Step 5 — Dry-run (mandatory in non-emergency scenarios)
+### Step 5 — Pre-restore validation (mandatory in non-emergency scenarios)
+
+> **Velero has no `--dry-run` flag on `velero restore create`.** Earlier guidance that suggested one was incorrect. Validate via `velero backup describe`, then trial-restore on a non-production cluster.
 
 ```bash
-velero restore create myapp-restore-dryrun \
+# 1. Inspect what the backup actually contains, including resource and PV counts
+velero backup describe myapp-backup-20260501 --details
+
+# 2. Trial-restore on a non-production cluster, then inspect the result
+velero restore create myapp-restore-trial \
   --from-backup myapp-backup-20260501 \
   --include-namespaces myapp-production \
-  --dry-run -o yaml
+  --wait
+velero restore describe myapp-restore-trial --details
+velero restore logs myapp-restore-trial
 ```
 
-Review the dry-run output for:
-- Unexpected resource counts (compare against last-known production state)
-- Resources that would be overwritten if `existingResourcePolicy: update`
-- Missing PV restore entries
+Review the validation output for:
+- Unexpected resource counts (compare `velero backup describe` against last-known production state).
+- Resources that would be overwritten if `existingResourcePolicy: update`.
+- Missing PV restore entries (check `restorePVs` and the volume snapshot list).
+- Any partially-failed items in the trial-restore logs.
 
 ### Step 6 — Sign-off gate
 
@@ -196,7 +205,7 @@ Return:
 2. **Hard-stop assessment** — is this operation blocked? List exact rule triggered.
 3. **Evidence level** — live evidence, documentation-based, or inference
 4. **Approval status** — confirmed sign-off or pending
-5. **Recommended command** — dry-run first, then execute
+5. **Recommended command** — `velero backup describe --details` and a trial restore on a non-production cluster first, then execute on production
 6. **Rollback posture** — saved state file, re-apply command
 7. **Verification steps** — post-restore pod health, resource counts, PV binding status
 8. **Open risks** — hook coverage gaps, snapshot TTL mismatches, BSL credential posture