codex-genesis-harness 0.1.1 → 0.1.4

package/.codex/skills/genesis-release-orchestration/checklists/post-deployment-verification.md

@@ -0,0 +1,274 @@
+# Post-Deployment Verification Checklist
+
+**Purpose**: Verify deployment success and stability post-deployment  
+**Duration**: 10-15 minutes per stage  
+**Risk**: Critical - must verify before next canary stage or production acceptance  
+
+---
+
+## Section 1: Deployment Completion Verification (5 min)
+
+- [ ] **Deployment completed successfully**
+  - All pods/containers running: `kubectl get pods` shows all Ready
+  - New version deployed: Verify version in logs/status endpoint
+  - Previous version stopped: Old containers terminated
+  - Deployment time recorded: Under <30 min target
+  - No deployment errors: Check CI/CD logs for failures
+
+- [ ] **Database state consistent**
+  - Migrations completed: No pending migrations
+  - Data integrity: Sample data queries return expected values
+  - Rollback point saved: Previous database state available if needed
+  - No connection errors: Application connects to database successfully
+
+- [ ] **Configuration deployed correctly**
+  - Environment-specific config loaded: Correct database, API keys
+  - Feature flags set correctly: New features enabled/disabled as intended
+  - Secrets loaded: No "secret not found" errors in logs
+  - Logging configured: Logs at appropriate level (not DEBUG in prod)
+
+---
+
+## Section 2: Health Check Validation (5 min)
+
+- [ ] **Liveness checks passing**
+  - Endpoint: GET /health returns 200 OK
+  - Response time: <500ms
+  - Response body includes version: Confirm v2.5.0 deployed
+  - Check frequency: Every 10 seconds (configurable)
+
+- [ ] **Readiness checks passing**
+  - Endpoint: GET /ready returns 200 OK
+  - Database connection: Verified in response
+  - Cache connection: Verified (if used)
+  - External service connectivity: Verified
+  - All dependencies healthy
+
+- [ ] **Service discovery updated**
+  - Load balancer sees new instances: In rotation
+  - DNS resolves to new IPs: No stale DNS
+  - Service mesh updated: Istio/Envoy routes to new version
+  - No connection refused errors: Services can reach each other
+
+---
+
+## Section 3: Smoke Tests & Critical Workflows (10 min)
+
+- [ ] **Critical workflow #1: User authentication**
+  - Login successful: User can authenticate
+  - Session created: Token generated and valid
+  - Session persists: Token valid across requests
+  - Logout works: Session properly cleared
+
+- [ ] **Critical workflow #2: API response format** (if breaking changes)
+  - Response structure matches new format: { data: {...} } not { user: {...} }
+  - Required fields present: All documented fields in response
+  - Optional fields handled: Backward compatibility (if version allows)
+  - Error responses use new format: Consistent across all endpoints
+
+- [ ] **Critical workflow #3: Database writes & reads**
+  - Create operation works: New data persisted
+  - Read operation works: Can retrieve created data
+  - Update operation works: Data can be modified
+  - Delete operation works: Data removal works
+  - No data corruption: Integrity constraints honored
+
+- [ ] **Critical workflow #4: Third-party integrations**
+  - External API calls succeed: Payment processor, email service, etc.
+  - Webhooks received: Third parties can call back into service
+  - Error handling: Graceful failure if third party unavailable
+  - Retry logic: Automatic retries working for transient failures
+
+- [ ] **Critical workflow #5: Backward compatibility** (for minor versions)
+  - Old API endpoints still work: If not broken
+  - Old response format accepted: Where supported
+  - Deprecated endpoints return warning: Clear guidance to migrate
+  - Migration timeline respected: Old version still functional
+
+---
+
+## Section 4: Performance & Baseline Metrics (5 min)
+
+- [ ] **Response time metrics**
+  - P50 latency: Baseline ± 10% (healthy)
+  - P95 latency: Baseline ± 10% (acceptable)
+  - P99 latency: Baseline ± 20% (watch but acceptable)
+  - Median within SLA: <200ms for UI endpoints
+
+- [ ] **Error rate metrics**
+  - 5xx errors: <0.1% (critical threshold 1%)
+  - 4xx errors: Normal range (expected user errors)
+  - Timeout errors: <0.01%
+  - No error spike: Consistent with pre-deployment baseline
+
+- [ ] **Resource usage metrics**
+  - CPU usage: <70% (healthy, room for spike)
+  - Memory usage: <80% (healthy, no memory leaks)
+  - Disk usage: <80% (logs not filling disk)
+  - Network saturation: <60% (room for growth)
+
+- [ ] **Throughput metrics**
+  - Requests per second: Matching pre-deployment baseline
+  - No request queue buildup: Processing without delays
+  - Connections active: Stable count (not growing indefinitely)
+  - Cache hit rate: Expected % (if applicable)
+
+---
+
+## Section 5: Error Log Analysis (5 min)
+
+- [ ] **No critical errors**
+  - Exception rate: 0 or expected baseline
+  - Stack traces: Review any new errors (expected after deployment)
+  - Database errors: No connection failures or constraint violations
+  - Timeout errors: <0.01%
+
+- [ ] **No security alerts**
+  - Authentication failures: Normal rate (not spike)
+  - Authorization failures: Expected for denied access
+  - Suspicious patterns: No signs of attack or abuse
+  - Invalid requests: Normal rate (not DDoS)
+
+- [ ] **Error messages understandable**
+  - Error descriptions: Clear and actionable
+  - Error codes: Match documentation (if breaking changes)
+  - Stack traces: Available for engineering review
+  - Correlation IDs: Present for request tracing
+
+---
+
+## Section 6: Consumer Compatibility Check (5 min)
+
+- [ ] **Consumer health verified** (if breaking changes)
+  - Known clients connected: Monitoring shows active sessions
+  - No authentication errors: Clients authenticated successfully
+  - API response handling: Clients processing new format correctly
+  - Migration status: Clients on supported versions
+
+- [ ] **No consumer errors**
+  - Consumer-specific endpoints: Working (if any)
+  - Third-party clients: No connection refused
+  - Mobile apps: No crashes reported
+  - Web clients: Working across browsers
+
+- [ ] **Feedback mechanisms working**
+  - Error reporting: Clients can report issues
+  - Support channels: Accessible (email, Slack, support portal)
+  - Status page: Updated and accessible
+  - Incident escalation: Path clear if issues arise
+
+---
+
+## Section 7: Deployment Strategy Progression (5 min)
+
+**If Canary Deployment**:
+
+- [ ] **Stage decision made** (if canary, proceed to next stage?)
+  - Errors acceptable: <1% error rate (go)
+  - Performance stable: Within baseline (go)
+  - No critical issues: Team agrees safe to proceed (go)
+  - Consumer feedback positive: No complaints (go)
+
+  **Decision**:
+  - [ ] GO to next stage: 10% traffic
+  - [ ] PAUSE: Monitor additional 30 minutes
+  - [ ] ROLLBACK: Critical issue found, revert to previous version
+
+**If Blue-Green Deployment**:
+
+- [ ] **Switch decision made**
+  - All health checks passing on new (blue) environment
+  - Load balancer can switch: DNS/LB config tested
+  - Old (green) environment: Still running, ready for instant rollback
+  - Consumer requests: Ready to flow to new environment
+
+  **Decision**:
+  - [ ] SWITCH: Route traffic from green to blue
+  - [ ] PAUSE: Monitor additional 30 minutes
+  - [ ] ROLLBACK: Critical issue found, stay on green
+
+**If Rolling Deployment**:
+
+- [ ] **Next wave ready**
+  - Current wave: All healthy and verified
+  - Next wave: Ready for deployment
+  - Health checks: Configured and passing
+  - No errors in current wave: Safe to proceed
+
+  **Decision**:
+  - [ ] PROCEED: Deploy next wave
+  - [ ] PAUSE: Monitor additional 15 minutes
+  - [ ] ROLLBACK: Issue found, stop rolling deployment
+
+---
+
+## Section 8: Incident & Rollback Preparation (5 min)
+
+- [ ] **Rollback plan confirmed** (always keep ready)
+  - Previous version available: Docker image, config, DB snapshot
+  - Rollback steps documented: Clear procedure to revert
+  - Team trained: Everyone knows rollback procedure
+  - Estimated time: <5 minutes to rollback
+
+- [ ] **Incident response ready**
+  - Escalation path: Clear who to notify if issues
+  - On-call team: Available for next 1-2 hours
+  - Communication channels: Slack, PagerDuty, etc.
+  - Status page: Updated if incident occurs
+
+- [ ] **Monitoring alert thresholds**
+  - Error rate alert: >5% triggers page
+  - Latency alert: >2s P95 triggers page
+  - Resource alert: CPU >80% triggers warning
+  - Dependency alert: External service down triggers alert
+
+---
+
+## Red Flags - Consider Rollback
+
+🚨 **WARNING - Evaluate rollback**:
+
+- [ ] Error rate >1% (watch)
+- [ ] Latency spike >50% above baseline (watch)
+- [ ] Any 5xx errors in logs (investigate)
+- [ ] Database connection failures (investigate)
+- [ ] Consumer complaints (investigate)
+
+❌ **MUST ROLLBACK - Immediate action**:
+
+- [ ] Error rate >5% (ROLLBACK)
+- [ ] Complete service unavailability (ROLLBACK)
+- [ ] Data corruption detected (ROLLBACK)
+- [ ] Security vulnerability discovered (ROLLBACK)
+- [ ] Database rollback failed (CRITICAL - escalate)
+
+**If ROLLBACK needed**:
+1. Execute rollback procedure immediately
+2. Notify stakeholders: Team, consumers, support
+3. Investigate root cause
+4. Fix issue
+5. Re-test before attempting deployment again
+
+---
+
+## Sign-Off Template
+
+```
+DEPLOYMENT: v2.5.0 → Canary Stage 1 (1% traffic)
+DATE: 2026-05-31 14:30 UTC
+VERIFIED BY: [Name]
+DURATION: 1 hour monitoring
+
+Health checks: ✓ PASS
+Error rate: ✓ PASS (0.08%)
+Response time: ✓ PASS (baseline +5%)
+Critical workflows: ✓ PASS
+Consumers: ✓ PASS
+
+Issues found: None critical
+
+DECISION: ✅ PROCEED TO STAGE 2
+  Next: Deploy to 10% traffic
+  Monitor: 2 hours
+  Go/no-go decision: 16:30 UTC
+```

package/.codex/skills/genesis-release-orchestration/checklists/pre-release-validation.md

@@ -0,0 +1,220 @@
+# Pre-Release Validation Checklist
+
+**Purpose**: Verify release readiness before deployment approval  
+**Duration**: 10-15 minutes  
+**Risk**: Critical - must pass all checks for approval  
+
+---
+
+## Section 1: Version & Changelog Verification (5 min)
+
+- [ ] **VERSION file updated correctly**
+  - Current version in `/VERSION` matches git tag format (v{MAJOR}.{MINOR}.{PATCH})
+  - Version bump follows semantic versioning (breaking→major, feature→minor, patch→patch)
+  - No pre-release suffixes without approval (v2.5.0-rc1, etc.)
+
+- [ ] **Changelog entry exists**
+  - Entry in `SPEC_CHANGELOG.md` for this version
+  - Changelog includes: ✨Added, 🔄Changed, 🐛Fixed, ⚠️Deprecated, 🗑️Removed, 🔐Security
+  - All breaking changes prominently listed
+  - Migration guide links included for breaking changes
+  - Release date documented
+
+- [ ] **Git tags match release version**
+  - Tag format: `v{MAJOR}.{MINOR}.{PATCH}` (e.g., v2.5.0)
+  - Tag annotation includes breaking change summary
+  - Tag created on correct commit (HEAD of main branch)
+
+---
+
+## Section 2: Code Quality & Testing (5 min)
+
+- [ ] **Test coverage 80%+ verified**
+  - Run test suite: All tests pass
+  - Coverage report generated: 80%+ threshold met
+  - Critical paths have >90% coverage
+  - No skipped tests (`.skip()` calls removed)
+  - E2E tests passing (Phase 5 validation)
+
+- [ ] **No critical errors in logs**
+  - Linting passes: No errors, only warnings acceptable
+  - Type checking passes: All types resolved
+  - Security scan passing: No vulnerabilities found
+  - Dependency audit clean: No high-risk dependencies
+
+- [ ] **Build artifacts valid**
+  - Docker image builds successfully
+  - All layers optimized (no bloat)
+  - Image pushed to registry successfully
+  - SHA hash recorded for deployment traceability
+
+---
+
+## Section 3: Breaking Changes & Migration (10 min)
+
+- [ ] **All breaking changes documented**
+  - `SPEC_CHANGELOG.md` lists each breaking change:
+    * What changed (old vs new)
+    * Why it changed
+    * Consumer impact
+    * Migration deadline
+  - Count: N breaking changes documented
+
+- [ ] **Migration guides complete** (required if breaking changes >0)
+  - Guide for each breaking change with:
+    * Before/after code examples (3+ languages)
+    * Step-by-step migration instructions
+    * Common pitfalls section
+    * Troubleshooting FAQ
+    * Support contact info
+  - Migration guides linked in:
+    * Release notes
+    * API documentation
+    * Consumer communication template
+
+- [ ] **Affected consumers identified & notified**
+  - Identified: N clients/services affected
+  - Notified: Consumer list reviewed and approved
+  - Communication template prepared (email + Slack + in-app banner)
+  - Support team briefed on incoming migration questions
+
+- [ ] **Deprecation timeline clear** (for gradual migration)
+  - If using deprecation period:
+    * Current version: v2.x (old API still works)
+    * Deadline version: v3.0 (old API removed)
+    * Timeline: N months to migrate
+  - Example: "Old endpoint deprecated in v2.5, removed in v3.0 (6-month timeline)"
+
+---
+
+## Section 4: Deployment Readiness (5 min)
+
+- [ ] **Deployment runbooks prepared**
+  - Runbook exists for: dev, staging, production
+  - Each runbook includes:
+    * Pre-deployment steps (DB migrations, config validation)
+    * Deployment steps (build, push, deploy, restart)
+    * Post-deployment verification (health checks, smoke tests)
+    * Rollback procedure (trigger conditions, steps)
+  - Runbooks reviewed by ops team
+
+- [ ] **Health checks configured**
+  - Liveness probe configured: /health (returns 200)
+  - Readiness probe configured: /ready (checks dependencies)
+  - Metrics endpoint configured: /metrics (Prometheus format)
+  - Smoke test scenarios defined (3+ critical workflows)
+
+- [ ] **Database migrations prepared** (if applicable)
+  - Migration script exists and tested
+  - Backward compatible: Can rollback if needed
+  - Zero-downtime approach: Old code works during migration
+  - Data integrity verified: No data loss risk
+  - Estimated duration: <5 min migration window
+
+- [ ] **Configuration prepared**
+  - Config files generated for: dev, staging, prod
+  - Environment-specific values validated:
+    * Database URLs
+    * API keys / secrets (via secrets manager)
+    * Feature flags properly set
+    * Logging levels appropriate
+  - No hardcoded values found
+
+---
+
+## Section 5: Rollback Capability (5 min)
+
+- [ ] **Rollback plan tested**
+  - Rollback triggers defined: error rate >5%, latency >2s, health check fail
+  - Rollback steps documented and verified
+  - Previous version artifacts available: Docker image, config, DB state
+  - Rollback time: <5 minutes verified
+  - Rollback reverses all changes: code, config, DB state
+
+- [ ] **Deployment strategy matches risk**
+  - Risk score calculated: N/10
+  - Strategy selected:
+    * Low (1-2): Rolling deployment
+    * Medium (3-5): Blue-green deployment
+    * High (6-8): Canary (1%→10%→50%→100%)
+    * Critical (9-10): Scheduled deployment + manual approval at each stage
+  - Strategy team-reviewed and approved
+
+- [ ] **Monitoring configured**
+  - Dashboard created for deployment monitoring
+  - Alerts configured for error rate spike
+  - Alert thresholds set: >5% error rate = trigger alert
+  - Team on-call for deployment window
+  - Escalation path defined
+
+---
+
+## Section 6: Approvals & Sign-Off (5 min)
+
+- [ ] **Required approvals obtained**
+  - Risk score: N/10
+  - Approval required from:
+    * Low (1-2): Tech Lead (or auto-approve)
+    * Medium (3-5): Tech Lead + Lead Engineer
+    * High (6-8): Tech Lead + Product Lead
+    * Critical (9-10): CTO + scheduled window approval
+  - All approvers signed off with timestamp
+
+- [ ] **Deployment window scheduled**
+  - If breaking changes: scheduled deployment window
+  - Team availability: All on-call resources available
+  - Communication: Consumers notified of maintenance window (if needed)
+  - Rollback team: Same team on standby for 1 hour post-deployment
+
+- [ ] **Documentation complete**
+  - Release notes ready (version, what's new, what's fixed, breaking changes)
+  - Consumer communication sent (email + Slack + notification)
+  - Internal team briefed: engineering, support, ops
+  - External status page updated (if consumer-facing)
+
+---
+
+## Red Flags - STOP if any present
+
+❌ **MUST STOP - Do not proceed to deployment**:
+
+- [ ] Test coverage <80% (BLOCKER)
+- [ ] Breaking changes undocumented (BLOCKER)
+- [ ] Migration guides missing for breaking changes (BLOCKER)
+- [ ] Rollback plan untested (BLOCKER)
+- [ ] Required approvals missing (BLOCKER)
+- [ ] Version bump doesn't match semantic versioning (BLOCKER)
+- [ ] Critical errors in logs/build (BLOCKER)
+- [ ] Database rollback impossible (HIGH RISK)
+- [ ] Consumers not notified of breaking changes (HIGH RISK)
+
+**If STOP condition found**:
+1. Identify blocking issue
+2. Take corrective action
+3. Re-run validation checklist
+4. Do not deploy until all checks pass
+
+---
+
+## Sign-Off Template
+
+```
+RELEASE: v2.5.0
+DATE: 2026-05-31
+CHECKED BY: [Name]
+APPROVED BY: [Name] (Tech Lead)
+
+Version verification: ✓ PASS
+Test coverage: ✓ PASS (82%)
+Breaking changes documented: ✓ PASS (2 breaking changes)
+Deployment runbook: ✓ PASS
+Rollback tested: ✓ PASS
+Approvals obtained: ✓ PASS
+
+Risk Score: 6/10 (HIGH - canary deployment)
+Status: ✅ APPROVED FOR DEPLOYMENT
+
+Next step: Begin canary deployment (Stage 1: 1% traffic)
+Monitoring window: 1 hour
+Rollback team: On standby
+```

package/.codex/skills/genesis-release-orchestration/examples/example.md

@@ -0,0 +1,78 @@
+# Example: Orchestrating a Minor Version Release (v2.4.1 → v2.5.0)
+
+## Scenario
+
+The team has completed sprint work. SPEC_CHANGELOG.md shows 3 new optional features and 2 bug fixes — no breaking changes. A minor version release is required.
+
+## Trigger
+
+```
+/prep-release
+```
+
+Or auto-triggered when `VERSION` file is bumped and all tests pass.
+
+## Phase 1: Version Detection (2 min auto)
+
+```
+Current Version: v2.4.1
+Changes since last release:
+  - 0 BREAKING changes
+  - 3 FEATURE changes (new optional fields)
+  - 2 BUG_FIX changes
+
+Semantic versioning decision:
+  FEATURE changes → Minor version bump
+  Proposed: v2.4.1 → v2.5.0
+  Risk level: MEDIUM (new features, no breaking changes)
+```
+
+## Phase 2: Risk Assessment (3 min auto)
+
+```
+Risk score: 4/10 (MEDIUM)
+  - No breaking changes
+  - 3 new optional API fields (backward compatible)
+  - All 94 tests passing
+  - Recommended strategy: Blue-green deployment
+```
+
+## Phase 3: Runbook Generated (3 min auto)
+
+**`.docs/releases/v2.5.0-runbook.md`** created with:
+- Pre-deployment database migration steps (none needed)
+- Docker build + push commands for staging
+- Blue-green switch command
+- Health check endpoint list
+- Rollback command (instant switch back to Green)
+
+## Phase 4: Release Notes Generated (2 min auto)
+
+```markdown
+## v2.5.0 — May 31, 2026
+
+### Added
+- User profiles now include optional `avatarUrl` field
+- API supports `?include=preferences` query parameter
+- Notifications support `priority` field (low/medium/high)
+
+### Fixed
+- Fixed: User deletion no longer leaves orphaned sessions
+- Fixed: Rate limiter correctly resets after 1-hour window
+```
+
+## Approval
+
+Risk 4/10 → Lead Engineer approval required (not CTO level).
+
+```
+Approval: ✅ [Lead Engineer] @ 14:32 — LGTM
+Deployment: Cleared for blue-green deployment
+```
+
+## Outcome
+
+- ✅ Version bumped to v2.5.0
+- ✅ Runbook generated in 3 minutes
+- ✅ Release notes complete
+- ✅ Total prep time: ~10 minutes (vs 2+ hours manual)