@ccoalm/ccl-skills 0.1.0

package/dist/assets/marketplace/plugins/ccl-skills/skills/platform-service-connectivity/references/service-discovery-choice.md

@@ -0,0 +1,113 @@
+# Service Discovery: Registry vs k8s-Native (Pick One, or Mix)
+
+Service discovery is a platform choice, not a mandate. Both **registry-based** (Nacos / Consul / etcd / managed equivalent) and **k8s-native** (Service + EndpointSlices + pod labels) can support lane-based multi-env routing, canary, A/B, and shadow traffic. Pick by the team's actual needs.
+
+## The two modes side-by-side
+
+| Capability | Registry-based | k8s-native |
+|---|---|---|
+| In-cluster service lookup | SDK queries registry | k8s DNS resolves `<svc>.<ns>.svc.cluster.local` |
+| Lane filter | instance metadata tag | pod label `lane:` + VirtualService subset |
+| Per-developer dev lane | new instances registered with lane tag | new Deployment with lane label |
+| Canary % | per-instance weight OR VS weight | VS weight + DR subsets |
+| Header-match A/B | SDK filter + VS | VS headers.match |
+| Shadow / mirror | SDK custom | VS mirror |
+| Per-instance custom health (drain, weight-0 without delete) | YES — registry metadata mutable | NO — k8s readiness is binary |
+| Out-of-cluster client (laptop) direct lookup | YES — laptop SDK queries registry | NO — must use kubectl port-forward / VPN / Telepresence |
+| Multi-cluster federation | registry federation (Nacos / Consul cluster mesh) | Istio multi-cluster |
+| Cross-language SDK | requires per-language registry client | k8s client-go in Go; HTTP API in any language |
+| Bootstrap order | SD must exist before services (chicken-egg if SD itself needs SD) | k8s API is the bootstrap layer; everything depends on it anyway |
+| Operational burden | dedicated cluster (3+ replicas, MySQL backend, log cleanup, snapshot DR) | piggyback on existing k8s control plane |
+| Drift between registry truth and k8s truth | possible (registry says healthy, pod actually unscheduled) | impossible by definition |
+
+Both can do **multi-env parallel dev** and **online multi-env experiments**. The choice is about secondary capabilities and operational burden.
+
+## When registry-based is the better fit
+
+Pick registry-based SD when ≥ 2 of these are true:
+
+1. **Per-instance drain / weight tuning** matters operationally — e.g. you want to slowly remove a problematic replica from rotation without scaling down the Deployment.
+2. **Out-of-cluster clients are common** — laptop dev SDKs, CI runners, external test harnesses query for instance IPs directly without tooling intermediation.
+3. **Cross-cluster federation feels natural** with the registry's federation primitives and you've already invested in operating it.
+4. **Polyglot platform with mature registry SDK in each language** — Go, Java, Python all have first-class clients you trust.
+5. **Existing platform already runs the registry** — the migration cost outweighs the operational savings.
+
+## When k8s-native is the better fit
+
+Pick k8s-native SD when ≥ 2 of these are true:
+
+1. **Operational simplification** outweighs per-instance tuning needs — one fewer stateful service, no separate SDK to ship in every language.
+2. **Single source of truth** matters more than registry-level flexibility — drift between registry state and k8s state has burned the team before.
+3. **Laptop dev workflow tolerates** kubectl port-forward / Telepresence / VPN.
+4. **Istio multi-cluster** is acceptable for cross-cluster needs (or no multi-cluster need).
+5. **Greenfield platform** — no sunk cost in a registry to amortize.
+
+## Mixed / transitional mode
+
+Common pattern: **k8s-native for east-west service-to-service; registry-based for laptop / out-of-cluster access**.
+
+```
+In-cluster pod → in-cluster pod
+  → k8s Service DNS + VS subset routing
+  → fast, no SDK dependency on registry
+
+Laptop SDK → in-cluster pod
+  → query small read-only registry (sync'd from k8s)
+  → or use kubectl port-forward
+```
+
+The registry runs as a thin **k8s → registry sync** sidecar; only laptop/CI clients use it. In-cluster code uses k8s DNS. Reduces registry load drastically.
+
+## Migration paths
+
+### Adding registry (k8s-native → mixed)
+
+If you start k8s-native and later want laptop direct lookup:
+
+1. Deploy a small registry instance (3 replicas, modest sizing).
+2. Run a sync controller: watch k8s Service + EndpointSlices, write to registry.
+3. Give laptop SDKs the registry endpoint.
+4. In-cluster code unchanged — still uses k8s DNS.
+
+### Removing registry (registry-based → k8s-native)
+
+If your platform inherited registry-based SD and you want to drop it:
+
+1. **Inventory**: list every language SDK that uses the registry; every framework `withRegistry` / `WithResolver` configuration; every laptop tool that queries it.
+2. **Add k8s-native resolver** as a parallel resolver in the framework client.
+3. **Dual-resolve for a deprecation window**: query both, prefer k8s, fall back to registry, log discrepancies. Run for ≥ 1 release cycle.
+4. **Migrate laptop tools** to kubectl port-forward / VPN / Telepresence.
+5. **Decommission registry pods**, then SDKs, then the sync.
+
+Estimated work for a medium platform (~50 services, 3 languages): 1-2 quarters.
+
+## What this means for the platform skills
+
+The other references in this skill cover both modes:
+
+- `service-discovery-recipe.md` — registry-based concrete recipe (Nacos as the example).
+- `mesh-architecture.md` — mesh layer is independent of SD choice; works with either.
+- `rpc-framework-recipe.md` — RPC base.Request + ctx propagation work regardless of resolver type.
+- `multi-env-routing.md` — lane primitives use pod labels (k8s-native) AND/OR registry tags; the routing rules are the same.
+
+A platform team adopting this skill family picks the SD mode separately and reads the relevant reference accordingly. No skill mandates the registry choice.
+
+## Hard rules that apply to either mode
+
+1. Lane is propagated through every hop regardless of SD mechanism.
+2. Mesh handles mTLS / connection retry / transport timeout — SD is not in this path.
+3. Health is a real thing: registry heartbeat OR k8s readiness; either is fine, but it MUST exist and be honored at routing time.
+4. Bootstrap dependencies are documented: which component depends on which at startup. (k8s API → DNS → mesh control plane → app pods, OR k8s API → SD cluster → mesh control plane → app pods.)
+5. Out-of-cluster access has a documented path: registry direct, OR port-forward + VPN, OR ingress + lane header — choose at least one.
+
+## Verification
+
+- A new service registers (registry-mode) or has a k8s Service created (k8s-mode) within seconds of pod ready.
+- A lane-tagged request routes to lane-matching pods, and only to lane-matching pods.
+- A canary deploy shifts traffic via VS weight; verify access logs show the new ratio.
+- Killing one pod removes it from rotation within the health-check propagation window (≤ 15s typical).
+- An out-of-cluster client (laptop) can reach a service using the documented path; document the latency and any setup steps required.
+
+## Kubernetes Gateway API Baseline (v1.0 GA Oct 2023 → v1.5 Apr 2026)
+
+- **Gateway API v1.0 (per `kubernetes.io/blog/2023/10/31/gateway-api-ga/`, released Oct 2023) graduated Gateway / GatewayClass / HTTPRoute to v1 GA**, the replacement for the legacy `Ingress` API for new platform deployments. Subsequent releases: **v1.1 (May 2024)** graduated GAMMA (service mesh support) and **GRPCRoute** to Standard channel; **v1.5 (Apr 2026)** moved further experimental features to Stable per `kubernetes.io/blog/2026/04/21/gateway-api-v1-5/`. **Architecture choice for ingress / north-south traffic**: (a) **default to Gateway API HTTPRoute** for new clusters — explicit resource model (Gateway = listener + cert + class, HTTPRoute = routing rules) cleanly separates infrastructure-owner concerns from application-owner concerns; supports cross-namespace routing via `ReferenceGrant` (no more sharing namespaces just to reuse one ingress object); stable schema with GA backward-compat. (b) **keep legacy Ingress** only for clusters where the ingress controller has not yet adopted Gateway API or where downstream tooling (cert-manager Ingress-specific integration, monitoring dashboards keyed on `ingress.kubernetes.io/...` annotations) hasn't migrated; flag those as platform debt. **For service mesh GAMMA (east-west traffic)**: the same HTTPRoute resource configures east-west traffic policies, eliminating mesh-vendor-specific CRDs (Istio `VirtualService` / Linkerd `ServiceProfile`) for new policies — verify the team's mesh implementation supports GAMMA (Istio 1.24+ per ambient-GA cycle, Linkerd 2.16+, Cilium 1.16+) before standardizing on it. **Migration policy**: do not mass-migrate existing Ingress → Gateway API in one slice; convert per ingress as touched for other reasons, run a smoke test against the new HTTPRoute object before deleting the legacy Ingress, and verify cert-manager / monitoring / WAF tooling has Gateway API equivalents wired up. **Split-brain mitigation for clusters running both Ingress and Gateway API**: most modern ingress controllers (Istio Gateway, Envoy Gateway, NGINX, Contour, etc) support BOTH the legacy Ingress object AND Gateway API HTTPRoute. If two route objects (one Ingress, one HTTPRoute) bind to the same host or share an overlapping route, precedence is controller-specific and NOT standardized by Kubernetes — the team gets non-deterministic routing depending on the controller version and config. Required policy: declare **one route-type owner per host** at the platform level (either the host is Ingress-managed OR HTTPRoute-managed, never both), enforce via admission webhook or a CI check that scans for Ingress + HTTPRoute overlap on the same `hostnames`; during the per-ingress migration, delete the legacy Ingress in the same commit/PR that creates the HTTPRoute. Mixed ownership without a precedence rule is a P0 routing-correctness risk for production traffic.

package/dist/assets/marketplace/plugins/ccl-skills/skills/platform-service-connectivity/references/service-discovery-migration-playbook.md

@@ -0,0 +1,231 @@
+# Service Discovery Migration Playbook
+
+Step-by-step playbook for migrating service discovery between modes (registry-based ↔ k8s-native). For the strategic decision — when to migrate vs stay — see `service-discovery-choice.md`. This reference is the **how**.
+
+## Direction A: Registry-based → k8s-native (retiring the registry)
+
+When you've decided the registry's value no longer justifies its operational cost. Typical signals: operational burden (registry cluster + per-language SDKs + log cleanup), drift between registry truth and k8s truth, low usage of registry-only features (per-instance drain, off-cluster direct lookup).
+
+### Phase 0 — Inventory (1-2 weeks)
+
+Before touching code, build a complete dependency map.
+
+```
+1. List every framework client/server that uses the registry:
+     grep -rE "WithRegistry|WithResolver|nacos|consul|<registry-sdk-import>" <all-service-repos>
+2. List every laptop / CI tool that queries the registry directly:
+     ops team interview + grep CLI configs
+3. List every k8s cluster that runs registry pods:
+     kubectl get sts -A | grep <registry>
+4. List every cross-cluster federation config / SDK config:
+     check registry cluster config for federation peers
+5. Identify "registry-only" features in use:
+     per-instance drain — search for "weight: 0" set calls
+     per-instance metadata reads — search for instance.metadata access
+     off-cluster lookup — search laptop tools
+```
+
+Output: a checklist with one row per item, status `discovered / mapped / planned / migrated`.
+
+### Phase 1 — Add k8s-native resolver alongside (2-3 weeks)
+
+Make the framework client able to resolve via k8s without removing the registry path.
+
+```
+1. Implement k8s-native resolver (per language):
+     - Go: use kitex-contrib/registry-kubernetes OR custom k8s-client-based resolver
+     - Python: use kubernetes Python client to list Endpoints
+     - Other: equivalent k8s client
+2. Add as a SECOND resolver, not replacing the first.
+3. Make resolver choice a per-service-startup flag or env var:
+     RESOLVER_MODE = registry | k8s | dual
+   Default: registry (no behavior change yet).
+4. Test the k8s resolver against a non-prod lane; verify lane-tag routing works
+   via pod labels.
+5. Roll the new framework version to all services (no behavior change since
+   default mode is registry).
+```
+
+Risk: framework version bump must not break anyone. Make sure the new resolver code is exercised in tests but unreachable in default config until Phase 2.
+
+### Phase 2 — Dual-resolve for one release cycle (3-4 weeks)
+
+Run both resolvers in parallel, prefer k8s, fall back to registry, log discrepancies.
+
+```
+1. Flip RESOLVER_MODE to "dual" for one non-critical service group.
+2. Resolver behavior in dual mode:
+     a. Query k8s → result_k8s
+     b. Query registry → result_reg
+     c. Compare set-equal; if different, log warning with both sides.
+     d. Use result_k8s for actual routing.
+3. Watch metrics:
+     resolver_discrepancy_total{service=X}
+   Discrepancies are the gap — services in registry not in k8s, or vice versa.
+4. Investigate each discrepancy:
+     stale registry instances (registry doesn't know pod is gone) → registry health issue
+     missing k8s service/endpoints → service not registering correctly in k8s
+     timing skew during deploy → expected; should self-heal
+5. Once the discrepancy rate is < 0.1% sustained for 2 weeks, promote the
+   service group to k8s-only.
+6. Repeat per service group, criticality ascending: dev tools → batch → mid-tier → critical.
+```
+
+Risk: silent fallback to registry hides k8s misconfiguration. Make dual-mode discrepancies LOUD — page if discrepancy rate > 1%.
+
+### Phase 3 — Switch service registration off (per group, 2-4 weeks)
+
+When dual mode confirms parity, stop new services from registering to the registry.
+
+```
+1. Per service group, flip RESOLVER_MODE to "k8s" only.
+2. Remove server.WithRegistry(...) from framework default options for this group.
+3. Service no longer writes to registry; relies solely on k8s Service + EndpointSlices.
+4. Verify:
+     - cross-service calls still work (mesh + VS routing intact).
+     - off-cluster laptop tooling works via documented alternative
+       (kubectl port-forward / VPN / Telepresence / ingress + lane header).
+5. Keep registry cluster running; no clients write anymore, some may still read.
+```
+
+After all service groups migrated, no service writes to the registry. The registry still serves reads (for laptop / off-cluster tools using the old path).
+
+### Phase 4 — Migrate off-cluster tools (2-3 weeks)
+
+```
+1. Identify every off-cluster tool that queries the registry:
+     CLIs, IDE plugins, monitoring scripts, CI runners.
+2. For each tool, pick the replacement:
+     - Static config tools (point to one prod service) → use ingress + DNS
+     - Dev IDE / debug → kubectl port-forward instruction in docs
+     - Pipeline runners → control-plane API for any deploy ops; k8s API for lookups
+     - Internal monitoring → query k8s API or scrape via Prometheus
+3. Update tool docs; remove registry endpoint from tool configs.
+4. Ship updated tool versions; deprecate old versions over 1 release cycle.
+```
+
+Risk: a forgotten tool still queries the now-empty registry. Mitigation: keep registry running read-only for one extra cycle, monitor read traffic, alert on any source.
+
+### Phase 5 — Decommission registry (1 week)
+
+When registry read traffic from non-platform tools is zero for 2 weeks:
+
+```
+1. Audit-retention export: full immutable export of registry metadata + audit log
+   to long-term storage (object store with WORM lock, or equivalent).
+   Retention policy: 1 year minimum, OR longer if compliance / legal requires.
+   Owner: security + platform; signoff required before any deletion proceeds.
+2. Restore test: in a sandbox cluster, restore the export and verify queries
+   work. If restore is broken, fix it before proceeding.
+3. Scale registry StatefulSet to 0.
+4. Wait 1 week (real users surface gaps).
+5. Delete the StatefulSet + PVCs.
+6. Remove the registry MySQL backend (separate task; backup the DB first to the
+   same audit-retention storage).
+7. Remove infra-namespace YAML manifests for the registry.
+8. Archive registry-related code repositories (service_register, component_register,
+   etc.) — keep readable but archived; do not delete repos.
+9. Update platform onboarding docs: "Service discovery is k8s + Istio; no registry."
+10. Run a tabletop drill: simulate a new service deploy from scratch with no
+    registry knowledge required.
+```
+
+**Do NOT delete a registry without immutable audit export + restore test + security signoff.** Incident investigations 6+ months later may need historical service registration / federation state to reconstruct who-called-what. Once the registry is gone with no export, that evidence is unrecoverable.
+
+### Phase 6 — Reclaim resources
+
+- 3+ registry pods × N replicas (~6-9 GiB memory, 1.5-3 CPU)
+- MySQL backend (~10-50 GiB storage)
+- NAS storage for registry data + snapshots (~150 GiB across 3 replicas)
+- Cross-language SDK build + maintenance time (~1-2 engineer-weeks/quarter)
+- Daily log cleanup CI (~1.9 GB/day if naive — now zero)
+
+### Total timeline
+
+For a medium platform (~50 services, 3 languages, 1 cluster):
+
+| Phase | Duration | Risk |
+|---|---|---|
+| 0. Inventory | 1-2 weeks | low |
+| 1. Add k8s resolver | 2-3 weeks | medium (framework change) |
+| 2. Dual-resolve burn-in | 3-4 weeks | medium (discrepancies surface) |
+| 3. Switch registration off | 2-4 weeks | low-medium (per group, gradual) |
+| 4. Off-cluster tool migration | 2-3 weeks | low (tools updated incrementally) |
+| 5. Decommission | 1 week | low |
+| **Total** | **~3 months active work + 1 month observation** | |
+
+For a large platform (~500 services, polyglot, multi-cluster), budget 2x.
+
+### Rollback path (within migration)
+
+If discrepancies in Phase 2 are too high to clear, OR off-cluster tools cannot be migrated in time:
+
+```
+1. Flip RESOLVER_MODE back to "registry" (default in Phase 1).
+2. Re-enable server.WithRegistry(...) if it was removed.
+3. Keep registry pods running; do not decommission.
+4. Document the cause; address in a follow-up cycle.
+```
+
+Rollback is cheap if you stop before Phase 5 (decommission). After Phase 5, restoring is a fresh registry install.
+
+## Direction B: k8s-native → Add a thin registry (mixed mode)
+
+For platforms that started k8s-native and now want off-cluster direct lookup without the operational cost of a full registry-of-record.
+
+```
+1. Deploy a small registry (e.g. 3 replicas, low resource ask).
+2. Run a sync controller (typically ~200 LOC) that watches k8s
+   Service + EndpointSlices and writes corresponding entries to the registry.
+3. The registry is NOT the source of truth; k8s is. The registry is a
+   read-only mirror.
+4. Off-cluster tools query the registry; no in-cluster code reads it.
+5. Service registration: services do NOT explicitly register; the sync
+   controller picks them up from k8s.
+```
+
+Pros: in-cluster routing unchanged (k8s + Istio); off-cluster tools get easy lookups.
+Cons: an additional system to operate, even if thin.
+
+Skip this entirely if off-cluster lookups can be satisfied by kubectl port-forward / VPN / Telepresence.
+
+## Verification rituals (both directions)
+
+After each phase, run:
+
+```
+1. Synthetic deploy: deploy a new dev-lane service; verify it's reachable
+   end-to-end from at least one client in each migrated mode.
+2. Pod kill: kill one pod of a multi-replica service; verify routing skips
+   it within the health check propagation window.
+3. Lane routing: send requests with lane header; verify they land on
+   lane-matching pods only.
+4. Off-cluster reach: from a laptop with the documented path, reach an
+   in-cluster service; verify latency is acceptable.
+5. Discrepancy scan (dual mode only): resolver_discrepancy_total{} = 0
+   or below tolerance threshold for the past 24h.
+```
+
+## Anti-patterns during migration
+
+- **Removing the registry path before dual-resolve burn-in clears**: causes silent breakage when k8s and registry disagree.
+- **Migrating critical services first**: any framework bug surfaces in the riskiest place. Migrate dev tools and batch first.
+- **Skipping the discrepancy investigation**: warnings get ignored, you don't actually achieve parity, then production breaks.
+- **Decommissioning registry too eagerly**: a forgotten off-cluster tool fails silently. Keep registry running read-only for at least one full release cycle past Phase 3.
+- **Trying to migrate in one big-bang release**: there is no big-bang here. Per-service-group gradual rollout is the only safe path.
+- **Mixing migration with framework refactoring**: each is risky on its own. Don't bundle.
+
+## Outcome
+
+After Direction A:
+- Single source of truth (k8s API).
+- One fewer stateful service to operate.
+- Per-language SDKs slim down (drop registry client dependency).
+- Bootstrap order simplifies (no SD chicken-egg).
+- Off-cluster access goes through documented tooling, not direct lookup.
+- Per-instance custom health states (drain, weight-0-without-delete) are lost — accept or design alternatives (e.g. labeled "drain" pod with NetworkPolicy excluding it from traffic).
+
+After Direction B:
+- In-cluster routing unchanged.
+- Off-cluster lookup becomes easy.
+- Thin sync controller adds one moving part; keep it simple.

package/dist/assets/marketplace/plugins/ccl-skills/skills/platform-service-connectivity/references/service-discovery-recipe.md

@@ -0,0 +1,131 @@
+# Service Discovery Recipe
+
+Registry-based SD. Nacos is the recurring example; rules generalize to Consul, etcd, ZooKeeper, or a managed equivalent.
+
+## Why registry, not k8s Service DNS
+
+k8s Service DNS gives you "an IP for `<service>.<ns>.svc.cluster.local`". Registry gives you:
+- Per-instance lane / env / region tags.
+- Weighted load balancing.
+- Custom health states (e.g. drain mode).
+- Cross-cluster federation (one logical service, multiple k8s clusters).
+- Lane-based subsetting that the SDK resolver consumes.
+
+If you only have k8s DNS, lane-based routing must live entirely in the mesh, and SDK-level weighting/draining is impossible. For non-trivial multi-env setups, that constraint hurts.
+
+## Topology
+
+```
+   ┌──────────────────────────────────────────────┐
+   │  Registry cluster (e.g. Nacos), N replicas   │
+   │  - StatefulSet, headless service             │
+   │  - NOT injected into mesh                    │
+   │  - Backing store: MySQL or embedded raft     │
+   │  - Anti-affinity, multi-AZ                   │
+   └──────┬───────────────────────────────────────┘
+          │ register / heartbeat / query
+   ┌──────┴───────────────────────────────────────┐
+   │                                              │
+   ▼                                              ▼
+┌──────────────┐                          ┌──────────────┐
+│ Server pod   │                          │ Client pod   │
+│ - Framework  │  on start: register      │ - Framework  │
+│   server     │  service+lane+addr       │   client     │
+│   suite      │  on shutdown: deregister │   resolver   │
+│ - Heartbeat  │                          │ - Caches      │
+│   ttl        │                          │   instances  │
+└──────────────┘                          │ - Subsets by │
+                                          │   lane tag   │
+                                          └──────────────┘
+```
+
+## Instance metadata (registration)
+
+When a service registers, it MUST include:
+
+```yaml
+service_name: <owner>.<class>.<env>           # PSM-style; or service.name OTel attr
+ip: <pod IP>
+port: <service port>
+tags:
+  lane: <lane label>
+  idc: <region/data center>
+  cluster: <k8s cluster name>
+  version: <build/image tag>
+  weight: <int, default 100>
+ephemeral: true                                # auto-deregister on TTL miss
+healthy: true
+```
+
+Missing `lane` tag = lane routing broken for callers.
+
+## Healthcheck contract
+
+- Service exposes a `/healthz` or `/ping` endpoint that returns 200 only when ready to serve.
+- Registry polls or accepts heartbeat (Nacos uses heartbeat).
+- Heartbeat interval: 5s; TTL: 15s. Unhealthy instances drop within 15s.
+- During pod terminationGracePeriod: deregister first, then drain in-flight requests, then exit. Pre-stop hook is the standard mechanism.
+
+## Resolver behavior in SDK
+
+The framework client resolver:
+1. Calls registry on first connect: `select instances of <callee> with lane=<my-lane>`.
+2. If 0 instances for current lane, fall back to a documented lane (e.g. `prod` or `default`). Make the fallback rule explicit; "silently fall through" is a bug breeder.
+3. Caches instance list with refresh interval (10–30s).
+4. Subscribes to registry change events (push) where the protocol supports it.
+5. Filters out unhealthy instances per registry-reported state.
+6. Load-balances within filtered list (round-robin, weighted, or P2C).
+
+## Cross-language registration
+
+If services span Go, Python, Java, Node: every language SDK must agree on:
+- Service name shape (lowercase, dot-separated, no underscores if gRPC is in scope — see `grpc-authority-workaround.md`).
+- Tag key names (`lane`, not `env`; pick one).
+- Heartbeat interval and TTL.
+- Health state semantics.
+
+Document the cross-language contract; do not let each language SDK reinvent.
+
+## Dev / staging / prod differences
+
+| Env | Registry use | Reason |
+|---|---|---|
+| Local dev | optional; SDK can fall back to direct address from config | dev shouldn't depend on shared registry |
+| Per-developer lane in shared cluster | required | callee's lane tag is how mesh routes |
+| Staging / pre-prod | required | parity with prod |
+| Prod | required | obvious |
+
+Provide a `LOCAL_HOST=...` env var or equivalent for local dev. The SDK switches resolution mode based on env detection.
+
+## Registry sizing
+
+- 3 replicas minimum for HA.
+- StatefulSet with persistent backing (MySQL, raft, or built-in).
+- CPU/mem: start with 500m / 1Gi requests and 50Gi PVC per member; tune by instance count, heartbeat volume, and retained audit/history.
+- Anti-affinity across nodes and AZs.
+- Use a headless Service; set `publishNotReadyAddresses: true` only when the registry needs peer bootstrap before readiness.
+- `podManagementPolicy: Parallel` can shorten bootstrap for peer-forming registries, but only if the registry tolerates parallel starts.
+- DO NOT mesh-inject. Registry bootstraps the mesh; mesh cannot depend on registry that needs mesh to talk.
+- If the registry emits high-volume access / raft / change-history logs, attach a log-cleaner sidecar scoped to the registry log subdirectory. Do not mount shared host log roots; use an allowlist and deletion metric.
+
+## Registry snapshot / DR discipline
+
+- Keep immutable registry metadata exports or snapshots on a separate backup path; incident reconstruction can need historical service registration and federation state.
+- Snapshot restore is a whole-cluster DR action, not a single-replica repair. Gate restore behind an explicit operator-set DR marker and require the registry StatefulSet to be scaled to 0 first.
+- For a single failed member in a healthy quorum, use the registry's native member remove / add or equivalent replacement flow; do not restore an old snapshot into a live quorum.
+- Before decommissioning a registry, run a restore test in a sandbox cluster and verify representative service lookup queries work.
+
+## Failure modes
+
+- **Registry briefly unavailable**: SDK uses cached instances for a bounded TTL (e.g. 5 min). Beyond that, request fails — better than calling a dead pod.
+- **Stale instances** (registry crashed without cleanup): readiness probe + heartbeat TTL should converge within minutes. Manual cleanup tool needed for outliers.
+- **Split brain across DCs**: registry federation; reads local, writes propagated. Local-first reads avoid cross-DC latency.
+
+## Migration off another SD mechanism
+
+If migrating from k8s DNS or another registry, run both side-by-side for a deprecation window:
+- Dual-write: services register to both.
+- Dual-read: SDK reads from new registry; falls back to old if new returns empty for a service.
+- Burn-in: track which services use which path (per-call metric).
+- Cutover: stop dual-write per service group once new is proven.
+- Tear-down: remove old SDK paths after one full release cycle clean.