tribunal-kit 2.4.6 → 3.0.0

package/.agent/skills/deployment-procedures/SKILL.md

@@ -1,188 +1,145 @@
----
-name: deployment-procedures
-description: Production deployment principles and decision-making. Safe deployment workflows, rollback strategies, and verification. Teaches thinking, not scripts.
-allowed-tools: Read, Write, Edit, Glob, Grep
-version: 1.0.0
-last-updated: 2026-03-12
-applies-to-model: gemini-2.5-pro, claude-3-7-sonnet
----
-
-# Deployment Principles
-
-> Deployments are not risky because of the code. They are risky because of all the
-> assumptions that have never been tested in production.
-
----
-
-## The Core Tension
-
-Speed vs. safety. Moving fast reduces iteration time. Moving carefully reduces incidents.
-The answer is not "always be careful" — it's **make fast safe**.
-
-That means:
-- Deployments that are reversible
-- Changes that are observable in real time
-- Failures that are isolated to a subset of users
-- State changes that can be undone without code changes
-
----
-
-## Five Phases of Safe Deployment
-
-### Phase 1 — Pre-Flight
-
-Before touching anything in production:
-
-- [ ] Tests passing on the branch being deployed
-- [ ] No pending schema migrations that will break the current production code
-- [ ] Feature flags in place for any risky changes
-- [ ] Rollback plan confirmed — "delete the feature flag" is a valid plan, "redeploy" is not (too slow)
-- [ ] Team notified if deployment will cause visible disruption
-
-### Phase 2 — Database First
-
-If there are schema changes:
-
-- Deploy the migration **before** the code that depends on it
-- Verify the migration completed and the database is healthy
-- The new code must be backward-compatible with the old schema (for the window during which old pods are still running)
-
-**Never:**
-- Add NOT NULL without a DEFAULT in the migration
-- Drop a column in the same deployment that removes the code referencing it
-- Run a migration that locks the table for more than a few seconds without scheduling a maintenance window
-
-### Phase 3 — Code Deploy
-
-Deploy with traffic distribution:
-
-| Strategy | Risk | When to Use |
-|---|---|---|
-| Direct (all-at-once) | High | Small teams, low traffic, with immediate rollback |
-| Rolling | Medium | Multiple instances, gradual update, auto-rollback on health fail |
-| Blue/Green | Low | Mission-critical services, instant switch and rollback |
-| Canary | Very low | Unknown risk level, expose to 1–5% of traffic first |
-
-### Phase 4 — Verify
-
-After deploying, watch:
-
-- Error rate — compare to pre-deploy baseline, not zero
-- Response time P50, P95, P99 — not just average
-- Business metric if visible (conversion, checkout completion)
-- Key logs for new error patterns
-
-Wait at minimum:
-- 5 minutes for canary verification
-- 15 minutes for a rolling deploy
-- Until traffic covers the full daily pattern for any significant feature
-
-### Phase 5 — Complete or Roll Back
-
-**Roll back when:**
-- Error rate increases by more than 2x pre-deploy baseline
-- P95 latency increases significantly without an expected cause
-- A critical user path stops working
-
-**Complete when:**
-- All metrics stable for the required observation window
-- All instances updated
-- Feature flags cleaned up if used
-
----
-
-## Rollback vs. Roll Forward
-
-| Scenario | Recommendation |
-|---|---|
-| Bug in new code, no data mutations | Roll back (redeploy previous version) |
-| Bug in new code, data already mutated | Roll forward (fix the mutation in a follow-up deploy) |
-| Schema migration caused the issue | Fix forward — migrations are rarely safely reversible |
-| Feature flag controls the issue | Turn off the flag — fastest rollback possible |
-
----
-
-## Environment Hierarchy
-
-Code flows one direction: dev → staging → production. Never skip staging for anything non-trivial.
-
-- **Development:** Fast iteration, local data, no external consequences
-- **Staging:** Production-like data (anonymized), used for final verification
-- **Production:** Real users, real consequences, thorough before touching
-
----
-
-## What a Deployment Runbook Contains
-
-For any significant deployment, document before starting:
-
-```
-Date/Time:         
-Engineer:          
-What is changing:  
-Why:               
-Expected behavior: 
-How to verify:     
-Rollback plan:     
-Time to rollback:  
-```
-
----
-
-## Output Format
-
-When this skill produces a recommendation or design decision, structure your output as:
-
-```
-━━━ Deployment Procedures Recommendation ━━━━━━━━━━━━━━━━
-Decision:    [what was chosen / proposed]
-Rationale:   [why — one concise line]
-Trade-offs:  [what is consciously accepted]
-Next action: [concrete next step for the user]
-─────────────────────────────────────────────────
-Pre-Flight:  ✅ All checks passed
-             or ❌ [blocking item that must be resolved first]
-```
-
-
-
----
-
-## 🤖 LLM-Specific Traps
-
-AI coding assistants often fall into specific bad habits when dealing with this domain. These are strictly forbidden:
-
-1. **Over-engineering:** Proposing complex abstractions or distributed systems when a simpler approach suffices.
-2. **Hallucinated Libraries/Methods:** Using non-existent methods or packages. Always `// VERIFY` or check `package.json` / `requirements.txt`.
-3. **Skipping Edge Cases:** Writing the "happy path" and ignoring error handling, timeouts, or data validation.
-4. **Context Amnesia:** Forgetting the user's constraints and offering generic advice instead of tailored solutions.
-5. **Silent Degradation:** Catching and suppressing errors without logging or re-raising.
-
----
-
-## 🏛️ Tribunal Integration (Anti-Hallucination)
-
-**Slash command: `/review` or `/tribunal-full`**
-**Active reviewers: `logic-reviewer` · `security-auditor`**
-
-### ❌ Forbidden AI Tropes
-
-1. **Blind Assumptions:** Never make an assumption without documenting it clearly with `// VERIFY: [reason]`.
-2. **Silent Degradation:** Catching and suppressing errors without logging or handling.
-3. **Context Amnesia:** Forgetting the user's constraints and offering generic advice instead of tailored solutions.
-
-### ✅ Pre-Flight Self-Audit
-
-Review these questions before confirming output:
-```
-✅ Did I rely ONLY on real, verified tools and methods?
-✅ Is this solution appropriately scoped to the user's constraints?
-✅ Did I handle potential failure modes and edge cases?
-✅ Have I avoided generic boilerplate that doesn't add value?
-```
-
-### 🛑 Verification-Before-Completion (VBC) Protocol
-
-**CRITICAL:** You must follow a strict "evidence-based closeout" state machine.
-- ❌ **Forbidden:** Declaring a task complete because the output "looks correct."
-- ✅ **Required:** You are explicitly forbidden from finalizing any task without providing **concrete evidence** (terminal output, passing tests, compile success, or equivalent proof) that your output works as intended.
+---
+name: deployment-procedures
+description: Production application deployment mastery. Zero-downtime deployment strategies (Blue/Green, Rolling updates), Container orchestration (Docker/ECS), CI/CD pipelines, secrets injection, database migration safety, health checks, and rollback contingencies. Use when moving code from development to production execution.
+allowed-tools: Read, Write, Edit, Glob, Grep
+version: 2.0.0
+last-updated: 2026-04-02
+applies-to-model: gemini-2.5-pro, claude-3-7-sonnet
+---
+
+# Deployment Procedures — Production Execution Mastery
+
+> Code on a laptop delivers zero value. Shipping is a feature.
+> Deployments should be boring, predictable, and 100% automated. Manual execution is a vulnerability.
+
+---
+
+## 1. Zero-Downtime Deployment Strategies
+
+Stopping a server, pulling code, building, and restarting is unacceptable. This results in 30-120 seconds of 502 Bad Gateway errors.
+
+### Blue/Green Deployment
+- Two identical environments (Blue is live, Green is idle).
+- Deploy v2 to Green. Run smoke tests on Green.
+- Swap the reverse proxy (Nginx or Load Balancer) router from Blue to Green.
+- Zero downtime. Rollback is instant (swap router back to Blue).
+
+### Rolling Updates (Container Clusters)
+- If you have 5 containers running v1. 
+- Spin up 1 container running v2. Wait for it to pass health checks.
+- Drain and terminate 1 container of v1.
+- Repeat until all 5 containers run v2.
+
+```bash
+# Docker Swarm / ECS / Kubernetes inherently handle rolling updates
+docker service update --image myapp:v2 --update-parallelism 1 --update-delay 10s myapp_web
+```
+
+---
+
+## 2. Infrastructure as Code (IaC) CI Pipelines
+
+All deployment logic must be codified and checked in alongside the application code.
+
+```yaml
+# .github/workflows/deploy.yml
+name: Production Deploy
+
+on:
+  push:
+    branches: [ "main" ]
+
+# Concurrency limits prevent race conditions if two commits are pushed rapidly
+concurrency: 
+  group: production-deploy
+  cancel-in-progress: true
+
+jobs:
+  build_and_deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      
+      # 1. CI Phase: Fast fail
+      - name: Install & Audit
+        run: npm ci && npm audit --audit-level=high
+      
+      - name: Unit Tests
+        run: npm test
+
+      # 2. Build Phase
+      - name: Build Assets
+        run: npm run build
+
+      # 3. CD Phase (Deployment via SSH/Docker)
+      - name: Deploy to Server
+        uses: appleboy/ssh-action@master
+        with:
+          host: ${{ secrets.SERVER_HOST }}
+          username: deploy_user
+          key: ${{ secrets.DEPLOY_SSH_KEY }}
+          script: |
+            cd /opt/myapp
+            git pull origin main
+            docker-compose up -d --build
+            # Container starts in background, port mapped to Nginx.
+```
+
+---
+
+## 3. Database Migration Safety Rules
+
+Schema changes cause 90% of severe deployment outages. 
+
+**The Expand-and-Contract Pattern (Zero Downtime DB Migrations):**
+Never drop columns or rename tables on a live system. Old code running against new schemas *will* crash.
+
+*Goal: Rename column `first_name` to `given_name`*
+- **Phase 1 (Expand):** Add `given_name` as a NEW, nullable column. The app writes to BOTH columns simultaneously, reads from `first_name`.
+- **Phase 2 (Migrate):** Run background script copying `first_name` data to `given_name`.
+- **Phase 3 (Swap):** Deploy v2 Application code that reads/writes exclusively to `given_name`.
+- **Phase 4 (Contract):** Drop the legacy `first_name` column weeks later.
+
+---
+
+## 4. The 5-Minute Rollback Guarantee
+
+If the new deployment throws persistent 5xx errors, how fast can you revert?
+If the answer relies on "recompiling the old git commit," you have failed.
+
+1. **Docker Tags:** Every build is tagged with the Git SHA (`myapp:a1b2c3d`). Reverting is a split-second container swap.
+2. **Feature Flags:** The code deployed completely dormant. If it breaks when toggled via flag, the rollback is hitting the "Off" button on LaunchDarkly (Zero code deployed).
+3. **Database Integrity:** Migrations are explicitly atomic (`BEGIN; DROP TABLE...; COMMIT;`) so failures roll back seamlessly.
+
+---
+
+## 🤖 LLM-Specific Traps (Deployments)
+
+1. **The `git pull && pm2 restart` Trap:** AI defaults to suggesting raw SSH into a VPS, running `git pull`, and manually restarting the daemon. This guarantees downtime, unrepeatable builds, and ignores multi-node infrastructure.
+2. **Storing Secrets in GitHub Code:** Embedding `.env.production` heavily into the deployment pipeline instead of exclusively using GitHub Secrets/AWS Parameter Store injection mapping.
+3. **Missing Health Checks:** Deploying containers without explicitly defining a `/healthz` heartbeat, meaning the orchestrator will blindly route traffic to unbooted API instances.
+4. **Destructive Migrations:** Recommending `npx prisma db push` (destructive) in production instead of `npx prisma migrate deploy` (tracked, safe).
+5. **Node Modules Cache Bloat:** Downloading 800MB of `node_modules` repeatedly inside CI jobs without properly leveraging GitHub Actions Cache, doubling execution execution limits.
+6. **Deploying Untested Code:** Writing deployment workflows that jump straight to the build/push phase, skipping the mandatory Lint/TypeCheck/Test safety pipeline sequence.
+7. **Race Conditions:** Failing to enforce `concurrency: cancel-in-progress` in CI strings, resulting in Commit B deploying before Commit A under chaotic PR merging circumstances.
+8. **Blind SSH Keys:** Generating GitHub Action files relying on SSH but forgetting to explicitly add `StrictHostKeyChecking no` configuration, making the pipeline hang forever at the server verification prompt.
+9. **Environment Discrepancy:** Building React/Vite payloads locally on MacOS and `scp`ing the static files via ZIP upload, rather than enforcing isolated Linux Docker builds ensuring identical compilation architecture.
+10. **The Manual Verification Myth:** Generating workflows expecting human "click to deploy" buttons midway through CI loops when true CD should be reliably automated upon merging to target branches.
+
+---
+
+## 🏛️ Tribunal Integration
+
+### ✅ Pre-Flight Self-Audit
+```
+✅ Does the deployment strategy enforce Zero-Downtime rules (Blue/Green or Rolling)?
+✅ Are database schemas applying the 'Expand-and-Contract' non-destructive methodology?
+✅ Has the deployment architecture entirely eliminated raw `git pull` manual interventions?
+✅ Is the CI pipeline firmly enforcing Linting, Typing, and Testing sequences *prior* to image pushing?
+✅ Have catastrophic rollback pathways (e.g., reverting to explicitly tagged container SHAs) been defined?
+✅ Are production secrets injected dynamically via encrypted vaults/actions rather than statically defined?
+✅ Does the application expose a hardened `/healthz` endpoint for orchestration routers?
+✅ Is CI concurrency restricted to prevent multi-job deployment collision and overlap?
+✅ Has `npm ci` been enforced over the mutable `npm install` for deterministic build resolution?
+✅ Are structural builds occurring solely inside isolated Linux environments/runners (no localized SCPing)?
+```