ga-plugins-cli 0.1.0 → 0.1.1

package/plugins/migration-safety/skills/strangler-fig.md

@@ -0,0 +1,382 @@
+# Skill: strangler-fig
+
+Reference guide for the Strangler Fig migration strategy used at Garuda Airlines. This document is consumed by the `strangler-plan` command and is available to any agent or command in the migration-safety plugin.
+
+---
+
+## What Is the Strangler Fig Pattern?
+
+The Strangler Fig pattern (coined by Martin Fowler, named after a tropical plant that gradually replaces the host tree) is a strategy for incrementally migrating a legacy system to a new one. Instead of a big-bang rewrite — where the old system is replaced all at once — the Strangler Fig approach replaces the old system piece by piece, with both systems running in parallel until the old system is fully decommissioned.
+
+At any point during migration:
+- The **old system** (Java) is running and can serve 100% of traffic if the new system has problems.
+- The **new system** (Go) receives an increasing percentage of traffic as confidence grows.
+- A **gateway** (Kong, Nginx, ALB, Istio) routes traffic between the two systems, providing the control plane for the cutover.
+
+The key property: **rollback is always available** until the old system is decommissioned. This is the fundamental safety guarantee.
+
+---
+
+## Why Strangler Fig for the GA Java→Go Migration?
+
+The Garuda Airlines migration has specific characteristics that make Strangler Fig the right choice:
+
+1. **300 Java services → 30 Go services** (10:1 consolidation). A big-bang rewrite would require all 300 services to be ready simultaneously. Strangler Fig allows migration service-by-service, endpoint-by-endpoint.
+
+2. **Aviation criticality**. Booking, check-in, and loyalty services cannot tolerate extended outages. Strangler Fig keeps the Java service on standby, making rollback a 5-minute operation rather than a multi-hour recovery.
+
+3. **Long migration timeline**. A 10:1 consolidation at this scale will take months. Strangler Fig allows business value delivery (and technical de-risking) incrementally throughout the timeline.
+
+4. **Unknown unknowns**. Java/Spring Boot services often contain behaviors that are difficult to discover statically — timing-dependent behavior, caching effects, Spring AOP side effects. Running both systems in parallel and comparing production traffic reveals these discrepancies without outages.
+
+---
+
+## Migration Phases
+
+### Phase 1: Shadow Mode (Optional but Recommended)
+
+**What**: Go service receives a copy of every request (via gateway mirroring or dual-write) but its responses are discarded. Java service continues to serve all responses.
+
+**Purpose**: validate Go service correctness against real production traffic without any user impact.
+
+**When to use**: for high-risk services where even 5% canary traffic is too risky (e.g., payment processing, seat reservation).
+
+**Gateway implementation concept**:
+- Kong: `request-termination` + `http-log` to mirror to Go service
+- Nginx: `mirror` directive (`mirror /internal/go-mirror;`)
+- Istio: `mirror` field in VirtualService with `mirrorPercentage`
+- AWS ALB: not natively supported — use a sidecar proxy
+
+**Exit criteria**: Go service processes 24 hours of mirrored production traffic with zero errors logged.
+
+---
+
+### Phase 2: Canary (5% → 10%)
+
+**What**: a small percentage of real production traffic is routed to the Go service. Both services run in full production mode.
+
+**Purpose**: catch issues that shadow mode cannot reveal (e.g., Go service writes incorrect data to the shared DB, or produces responses that client apps cannot parse).
+
+**Duration**: minimum 30 minutes at 5%. Extend to 2-4 hours if the domain is critical.
+
+**Canary size decision**:
+
+| Traffic Sensitivity | Starting Canary % |
+|--------------------|--------------------|
+| Low (internal APIs, batch endpoints) | 10-20% |
+| Medium (customer-facing, non-transactional) | 5% |
+| High (booking, payment, seat assignment) | 1-2% |
+| Critical (real-time check-in, boarding gate) | Shadow mode first, then 1% |
+
+**Monitoring during canary** (see Monitoring section below).
+
+**Exit criteria**: canary percentage has been stable for the defined duration with error rate < 0.1% and latency within 20% of Java baseline.
+
+---
+
+### Phase 3: Ramp (25% → 50% → 75%)
+
+**What**: incremental traffic increase in stages, with observation periods between stages.
+
+**Ramp schedule** (adjust based on service criticality):
+- Low risk: 5% → 50% → 100% (skip intermediate steps)
+- Medium risk: 5% → 25% → 50% → 75% → 100%
+- High risk: 5% → 10% → 25% → 50% → 75% → 100% with 1-hour soak at each step
+
+**What to watch**: same metrics as canary, but also watch for load-dependent issues (connection pool exhaustion, GC pauses, Go goroutine leaks under sustained load).
+
+---
+
+### Phase 4: Full Cutover (100% Go)
+
+**What**: all traffic for the migrated endpoints routes to Go service. Java service runs but receives zero traffic.
+
+**Important**: this is NOT decommissioning. Java service stays running for the fallback window.
+
+**Fallback window**: minimum 2 weeks. This covers:
+- Business-cycle-specific bugs (e.g., a bug only triggered by monthly billing runs)
+- Long-tail error scenarios that did not appear at < 100% traffic
+- Edge cases discovered by customer support reports post-cutover
+
+---
+
+### Phase 5: Decommission
+
+**What**: Java service is stopped, infrastructure is released, gateway routing to Java is removed.
+
+**When**: after the fallback window with no rollback events.
+
+**This step is irreversible** (within the 2-week archive window). It should require explicit team sign-off.
+
+---
+
+## Anti-Corruption Layer (ACL) Pattern
+
+### When to Use It
+
+An Anti-Corruption Layer is needed when the Java and Go service have **different request or response shapes** for the same logical operation. This can happen when:
+
+- The 10:1 consolidation produces a Go service with a unified API that differs from the individual Java APIs
+- The Go service was designed with a cleaner schema (e.g., renamed fields, restructured JSON)
+- The Java service used non-standard date formats or snake_case vs. camelCase fields
+
+### What the ACL Does
+
+The ACL is a translation layer that sits between the gateway and the Go service (or between the client and the gateway). It:
+- Transforms incoming requests from the old (Java) schema to the new (Go) schema
+- Transforms outgoing responses from the new (Go) schema back to the old (Java) schema
+- Is transparent to the client — clients continue sending/receiving Java-compatible formats during the migration
+
+### Where to Implement It
+
+Options (choose one):
+
+| Location | Pros | Cons |
+|----------|------|------|
+| Gateway plugin (e.g., Kong transformer plugin) | No code in services; centralized | Gateway-specific; harder to test |
+| Sidecar on Go service | Service-independent; testable | More infrastructure |
+| Inside the Go service handler | Easiest to implement | Couples migration logic into product code |
+| Client-side versioned SDK | Clients control migration pace | Requires client coordination |
+
+**Garuda recommendation**: gateway plugin for simple field renames; sidecar for complex structural differences. Remove the ACL layer once all clients are updated to the new schema.
+
+### Removing the ACL
+
+Plan to remove the ACL after all clients are migrated to the new Go API. Do not leave the ACL permanently — it becomes a maintenance burden and a source of subtle bugs.
+
+---
+
+## Traffic Splitting Patterns by Gateway Type
+
+### Kong Gateway
+
+```yaml
+# Weighted upstream
+_format_version: "3.0"
+upstreams:
+  - name: booking-upstream
+    targets:
+      - target: booking-java-svc.internal:8080
+        weight: 95
+      - target: flight-ops-svc.internal:8081
+        weight: 5
+services:
+  - name: booking-service
+    host: booking-upstream
+    routes:
+      - name: booking-routes
+        paths: ["/api/v1/bookings"]
+```
+
+Adjust `weight` values at each ramp step.
+
+### Nginx
+
+```nginx
+# In upstream block:
+split_clients "${remote_addr}${request_uri}" $backend {
+    5%   go_upstream;
+    *    java_upstream;
+}
+
+upstream java_upstream { server booking-java-svc.internal:8080; }
+upstream go_upstream   { server flight-ops-svc.internal:8081; }
+
+location /api/v1/bookings {
+    proxy_pass http://$backend;
+}
+```
+
+Adjust the percentage in `split_clients` at each ramp step. Reload Nginx config: `nginx -s reload`.
+
+### AWS ALB
+
+Traffic weighting is done via weighted target groups on ALB listener rules:
+
+```json
+{
+  "Type": "forward",
+  "ForwardConfig": {
+    "TargetGroups": [
+      { "TargetGroupArn": "<JAVA_TG_ARN>", "Weight": 95 },
+      { "TargetGroupArn": "<GO_TG_ARN>",   "Weight": 5  }
+    ],
+    "StickinessConfig": {
+      "Enabled": false
+    }
+  }
+}
+```
+
+Update via: `aws elbv2 modify-rule --rule-arn <RULE_ARN> --actions <above JSON>`.
+
+### Istio / Envoy
+
+```yaml
+apiVersion: networking.istio.io/v1alpha3
+kind: VirtualService
+metadata:
+  name: booking-vs
+spec:
+  hosts:
+    - booking-svc
+  http:
+    - match:
+        - uri:
+            prefix: /api/v1/bookings
+      route:
+        - destination:
+            host: booking-java-svc
+            port: number: 8080
+          weight: 95
+        - destination:
+            host: flight-ops-svc
+            port: number: 8081
+          weight: 5
+```
+
+Apply: `kubectl apply -f booking-vs.yaml`. Update `weight` values at each ramp step.
+
+---
+
+## Fallback Strategy
+
+### How to Roll Back (At Any Phase)
+
+The rollback procedure is the same at every phase before decommissioning:
+
+1. **Update gateway**: route 100% of traffic back to Java service (same change as forward routing, reversed)
+2. **Verify Java is healthy**: check error rate and latency return to pre-migration baseline
+3. **Notify team**: post to incident channel, tag on-call
+4. **Do NOT restart the Go service** (keep it running for log inspection)
+5. **Write incident report**: what triggered rollback, what was the observed behavior
+
+Estimated rollback time: **< 5 minutes** for gateway-level rollback.
+
+### Decision Authority
+
+Any engineer in the on-call rotation can initiate rollback without management approval. Do not wait for a meeting — roll back first, investigate later.
+
+### After Rolling Back
+
+1. Collect Go service logs from the affected period
+2. Compare Go service behavior with Java service behavior (request/response logs)
+3. Write a bug report against the Go service
+4. Do not re-attempt cutover until the root cause is confirmed fixed and characterization tests are updated to cover the bug case
+
+---
+
+## Monitoring During Cutover
+
+### Metrics to Watch
+
+| Metric | Description | Alert Threshold |
+|--------|-------------|----------------|
+| HTTP 5xx error rate | Unhandled errors in Go service | > 0.1% of requests |
+| HTTP 4xx error rate | Client errors — watch for increases (may indicate schema mismatch) | > 20% increase from Java baseline |
+| p99 latency | 99th percentile response time | > 20% above Java baseline |
+| p50 latency | Median response time | > 10% above Java baseline |
+| Request throughput | Requests/second reaching Go service | < 10% of expected (may indicate routing failure) |
+| DB connection pool usage | Go service pool utilization | > 80% |
+| Go goroutine count | Active goroutines (Go-specific: indicates leaks) | > 2x expected per-RPS |
+| Business KPIs (domain-specific) | e.g., booking completion rate, seat availability accuracy | > 0.5% degradation |
+
+### Log Correlation
+
+During canary and ramp phases, both Java and Go services are processing requests. To correlate:
+- Use `X-Request-ID` header propagated from the gateway — both services should log this
+- Use distributed tracing (OpenTelemetry) if available — go-ga-lib should provide a tracing middleware
+- Compare error patterns: a spike in Go errors with no corresponding Java errors indicates a Go-specific bug
+
+### Grafana Dashboard Checklist
+
+Set up the following panels before starting cutover:
+
+- [ ] Go service error rate (5xx) over time
+- [ ] Java service error rate (5xx) over time (should remain flat)
+- [ ] Go vs. Java p99 latency side-by-side comparison
+- [ ] Traffic split percentage over time (shows canary/ramp progression)
+- [ ] Business KPI over time (domain-specific — e.g., "bookings created per minute")
+- [ ] Go service DB query duration
+- [ ] Go service goroutine count
+
+---
+
+## Data Migration Considerations
+
+### During Parallel Operation (Steps 1-5)
+
+If Go and Java services share the same database:
+- **Go writes, Java reads**: generally safe. Ensure Go does not write schemas incompatible with what Java expects (e.g., new NOT NULL columns with no default).
+- **Both writing to same table**: safe only if write operations are idempotent or there is no contention. If both services can write to the same row, implement optimistic locking.
+- **Go reads new DB schema, Java reads old**: use a compatibility view or migration approach that keeps the old columns populated.
+
+### Schema Migration Strategy
+
+1. **Expand**: add new columns/tables needed by Go service. Keep old columns. Both Java and Go work.
+2. **Migrate**: during ramp phase, backfill any data that Go service writes differently.
+3. **Contract** (only after Java decommissioned): remove old columns/tables that Java needed but Go does not.
+
+Never run the **Contract** step while Java is still running.
+
+### Eventual Consistency Sync
+
+If Go and Java services write to DIFFERENT databases (possible in the 10:1 consolidation):
+- Implement a sync job that copies Java DB writes to Go DB during the parallel period
+- Or implement dual-writes at the application level (write to both DBs for critical entities)
+- Remove the sync job immediately after Java decommissioning — it is a temporary migration artifact
+
+---
+
+## Common Pitfalls
+
+### 1. Session State in Java Services
+
+Java services using `HttpSession` or Spring Session store state in memory or in a session store (Redis, JDBC). Go services have no equivalent by default.
+
+**Risk**: users mid-session during a canary may have their session routed to the Go service, which cannot find the session.
+
+**Mitigation**: use sticky sessions at the gateway during migration (route a user to the same service for the duration of their session), then switch fully to Go. Alternatively, externalize session state before migration and ensure Go service reads from the same session store.
+
+### 2. Caching Inconsistencies
+
+Java services using `@Cacheable` with local caches (Caffeine, EhCache) have caches that are NOT shared with the Go service. During canary, a request going to Java may get a cached response while the same request going to Go hits the DB and gets a different (more current) response.
+
+**Mitigation**: use a shared distributed cache (Redis) before starting canary, so both Java and Go read/write the same cache. Or accept that canary will bypass Java caches and monitor for increased DB load.
+
+### 3. Distributed Transactions
+
+If a Java service starts a distributed transaction (XA, Saga, or CQRS event) and the Go service is responsible for part of the flow, there is no automatic coordination.
+
+**Mitigation**: do not cut over endpoints that participate in distributed transactions until all participants are ready to migrate together. Migrate the entire saga/workflow atomically, not endpoint by endpoint.
+
+### 4. Hidden Shared State in Java
+
+Java services with static fields, thread-local state, or Singleton caches that accumulate state across requests may behave differently under canary. For example: a Java Singleton that caches flight availability data will have a warm cache after hours of operation, while the Go service starts cold.
+
+**Mitigation**: identify all stateful caches (from ANALYSIS.md) and pre-warm Go service caches before canary, or account for cold-start behavior in the 5% canary phase.
+
+### 5. Time-Zone and Date Format Differences
+
+Java uses `ZonedDateTime` with explicit timezone handling; Go uses `time.Time` (always UTC by default). A subtle difference in timezone serialization (e.g., Java returns `"2024-03-15T14:30:00+07:00"` and Go returns `"2024-03-15T07:30:00Z"`) can break clients.
+
+**Mitigation**: add explicit serialization tests in characterization tests. Verify JSON date format matches between Java and Go responses.
+
+---
+
+## GA-Specific Notes: 10→1 Consolidation
+
+The Garuda Airlines migration consolidates 10 Java services into 1 Go service. The Strangler Fig pattern must be applied at the **endpoint group** level, not at the service level, because:
+
+- One Go service (`flight-ops-svc`) may absorb endpoints from 10 different Java services (`booking-java-svc`, `seat-java-svc`, `check-in-java-svc`, ...)
+- Each endpoint group from each Java service must go through its own canary → ramp → cutover cycle
+- The Go service can receive 100% of traffic for `POST /api/v1/bookings` while still receiving 0% of traffic for `POST /api/v1/check-in` (from a different Java service, not yet migrated)
+- The gateway must support endpoint-level traffic splitting, not just service-level
+
+**Implication for the strangler-plan command**: the `<endpoint-list>` parameter in each plan should cover only the endpoints being migrated in THIS phase, not all endpoints eventually destined for the Go service.
+
+**Consolidation cutover sequence suggestion**:
+1. Migrate the simplest Java service endpoints first (fewest dependencies, lowest traffic)
+2. Build confidence with the Go service under real production load before migrating complex endpoints
+3. Do not attempt full consolidation (all 10 Java services → 1 Go service) until at least 3 Java services have been successfully migrated and decommissioned
+4. The architecture team must own the consolidation sequencing — the migration-safety plugin does not decide which endpoints to migrate in which order