4runr-os 2.9.34 → 2.9.35

package/apps/gateway/DEPLOYMENT.md

@@ -0,0 +1,426 @@
+# Deployment Guide
+
+Complete guide for deploying the 4Runr Gateway to production.
+
+## Table of Contents
+
+1. [Prerequisites](#prerequisites)
+2. [Pre-Deployment Checklist](#pre-deployment-checklist)
+3. [Environment Setup](#environment-setup)
+4. [Database Setup](#database-setup)
+5. [Deployment Methods](#deployment-methods)
+6. [Post-Deployment Verification](#post-deployment-verification)
+7. [Rollback Procedures](#rollback-procedures)
+8. [Scaling](#scaling)
+
+## Prerequisites
+
+### Infrastructure Requirements
+
+- **PostgreSQL 12+**: For persistent storage
+- **Redis 6+**: For queue, rate limiting, and idempotency
+- **Docker & Docker Compose**: For containerized deployment (recommended)
+- **Node.js 18+**: For direct deployment (alternative)
+
+### Security Requirements
+
+- Strong database passwords
+- Secure JWT secrets (min 32 characters)
+- Encryption keys for sensitive data
+- SSL/TLS certificates for HTTPS
+- Firewall rules configured
+
+### Network Requirements
+
+- Port 3000: Gateway API (or configure reverse proxy)
+- Port 5432: PostgreSQL (internal only)
+- Port 6379: Redis (internal only)
+
+## Pre-Deployment Checklist
+
+- [ ] All environment variables configured
+- [ ] Database migrations tested
+- [ ] Secrets generated and stored securely
+- [ ] Health checks configured
+- [ ] Monitoring and alerting set up
+- [ ] Backup strategy in place
+- [ ] Rollback plan documented
+- [ ] Load testing completed
+- [ ] Security audit performed
+
+## Environment Setup
+
+### 1. Generate Secure Secrets
+
+```bash
+# Generate JWT secret
+openssl rand -base64 32
+
+# Generate encryption key
+openssl rand -base64 32
+
+# Generate API keys (if using memory mode)
+openssl rand -hex 32
+```
+
+### 2. Create Production Environment File
+
+```bash
+cd infra
+cp env.example .env.production
+```
+
+Edit `.env.production` with production values:
+
+```bash
+NODE_ENV=production
+GATEWAY_PERSISTENCE=postgres
+DATABASE_URL=postgresql://user:strong-password@db-host:5432/4runr_gateway
+REDIS_URL=redis://redis-host:6379
+
+AUTH_ENABLED=true
+AUTH_MODE=mixed
+JWT_SECRET=<generated-secret>
+JWT_ALG=HS256
+
+RATE_LIMIT_ENABLED=true
+RATE_LIMIT_MAX=100
+RATE_LIMIT_WINDOW_MS=60000
+
+SENTINEL_STORE_PLAIN=false
+SENTINEL_MODE=live
+SENTINEL_SHIELD_MODE=enforce
+
+LOG_LEVEL=warn
+```
+
+### 3. Store Secrets Securely
+
+**DO NOT** commit secrets to version control. Use:
+- Environment variable management (AWS Secrets Manager, HashiCorp Vault, etc.)
+- Docker secrets
+- Kubernetes secrets
+- CI/CD secret management
+
+## Database Setup
+
+### 1. Create Database
+
+```sql
+CREATE DATABASE 4runr_gateway;
+CREATE USER 4runr WITH PASSWORD 'strong-password';
+GRANT ALL PRIVILEGES ON DATABASE 4runr_gateway TO 4runr;
+```
+
+### 2. Run Migrations
+
+```bash
+cd apps/gateway
+pnpm db:generate
+pnpm db:migrate
+```
+
+Or via Docker:
+```bash
+docker-compose -f infra/docker-compose.prod.yml run --rm gateway pnpm db:migrate
+```
+
+### 3. Seed Initial Data (Optional)
+
+```bash
+# Seed API key
+pnpm seed:api-key
+```
+
+## Deployment Methods
+
+### Method 1: Docker Compose (Recommended)
+
+**Best for**: Single server deployments, small to medium scale
+
+1. **Build and start:**
+```bash
+cd infra
+docker-compose -f docker-compose.prod.yml up -d --build
+```
+
+2. **Verify services:**
+```bash
+docker-compose -f docker-compose.prod.yml ps
+docker-compose -f docker-compose.prod.yml logs -f gateway
+```
+
+3. **Check health:**
+```bash
+curl http://localhost:3000/health
+curl http://localhost:3000/ready
+```
+
+### Method 2: Kubernetes
+
+**Best for**: Large scale, high availability
+
+1. **Create namespace:**
+```bash
+kubectl create namespace 4runr-gateway
+```
+
+2. **Create secrets:**
+```bash
+kubectl create secret generic gateway-secrets \
+  --from-literal=jwt-secret='your-secret' \
+  --from-literal=database-url='postgresql://...' \
+  --namespace=4runr-gateway
+```
+
+3. **Deploy:**
+```bash
+kubectl apply -f k8s/gateway-deployment.yaml
+kubectl apply -f k8s/gateway-service.yaml
+```
+
+### Method 3: Direct Node.js
+
+**Best for**: Development, custom deployments
+
+1. **Build:**
+```bash
+cd apps/gateway
+pnpm install
+pnpm build
+```
+
+2. **Start:**
+```bash
+NODE_ENV=production node dist/index.js
+```
+
+Or use PM2:
+```bash
+pm2 start dist/index.js --name gateway
+pm2 save
+pm2 startup
+```
+
+## Post-Deployment Verification
+
+### 1. Health Checks
+
+```bash
+# Liveness
+curl http://your-server/health
+# Expected: {"status":"ok"}
+
+# Readiness
+curl http://your-server/ready
+# Expected: {"ready":true,"checks":{...}}
+
+# Startup
+curl http://your-server/startup
+# Expected: {"started":true}
+```
+
+### 2. API Functionality
+
+```bash
+# Create a run
+curl -X POST http://your-server/api/runs \
+  -H "x-api-key: your-key" \
+  -H "Content-Type: application/json" \
+  -d '{"name": "Test Run", "input": {}}'
+
+# List runs
+curl http://your-server/api/runs \
+  -H "x-api-key: your-key"
+
+# Check metrics
+curl http://your-server/metrics
+```
+
+### 3. Monitor Logs
+
+```bash
+# Docker
+docker-compose -f infra/docker-compose.prod.yml logs -f gateway
+
+# Kubernetes
+kubectl logs -f deployment/gateway -n 4runr-gateway
+
+# PM2
+pm2 logs gateway
+```
+
+### 4. Check Metrics
+
+Access Prometheus metrics at `/metrics` and verify:
+- HTTP request counts
+- Run creation/start/completion rates
+- Error rates
+- Queue job metrics
+- Rate limit hits
+
+## Rollback Procedures
+
+### Quick Rollback (Docker Compose)
+
+```bash
+cd infra
+
+# Stop current version
+docker-compose -f docker-compose.prod.yml down
+
+# Switch to previous version
+git checkout <previous-version-tag>
+
+# Rebuild and start
+docker-compose -f docker-compose.prod.yml up -d --build
+```
+
+### Database Rollback
+
+If migrations need to be rolled back:
+
+```bash
+cd apps/gateway
+pnpm prisma migrate resolve --rolled-back <migration-name>
+```
+
+**Warning**: Only rollback migrations if you're certain no data depends on them.
+
+### Configuration Rollback
+
+```bash
+# Restore previous environment file
+cp .env.production.backup .env.production
+
+# Restart services
+docker-compose -f docker-compose.prod.yml restart gateway
+```
+
+## Scaling
+
+### Horizontal Scaling
+
+The gateway is stateless and can be scaled horizontally:
+
+```bash
+# Docker Compose (multiple instances)
+docker-compose -f docker-compose.prod.yml up -d --scale gateway=3
+
+# Kubernetes
+kubectl scale deployment gateway --replicas=3 -n 4runr-gateway
+```
+
+### Load Balancing
+
+Use a load balancer (nginx, HAProxy, AWS ALB) in front of multiple gateway instances:
+
+```nginx
+upstream gateway {
+    least_conn;
+    server gateway1:3000;
+    server gateway2:3000;
+    server gateway3:3000;
+}
+```
+
+### Database Scaling
+
+- Use connection pooling (Prisma handles this)
+- Consider read replicas for read-heavy workloads
+- Monitor connection counts
+
+### Redis Scaling
+
+- Use Redis Cluster for high availability
+- Configure Redis Sentinel for failover
+- Monitor memory usage
+
+## Monitoring & Alerting
+
+### Key Metrics to Monitor
+
+- **HTTP Request Rate**: Requests per second
+- **Error Rate**: 4xx and 5xx errors
+- **Response Time**: p50, p95, p99 latencies
+- **Queue Depth**: Number of jobs waiting
+- **Database Connections**: Active connections
+- **Memory Usage**: Container/node memory
+- **CPU Usage**: Container/node CPU
+
+### Recommended Alerts
+
+- Error rate > 1%
+- Response time p95 > 1s
+- Queue depth > 1000
+- Database connections > 80% of max
+- Memory usage > 80%
+- Health check failures
+
+## Backup & Recovery
+
+### Database Backups
+
+```bash
+# Daily backup script
+pg_dump -h db-host -U 4runr 4runr_gateway > backup-$(date +%Y%m%d).sql
+
+# Restore
+psql -h db-host -U 4runr 4runr_gateway < backup-20231119.sql
+```
+
+### Configuration Backups
+
+```bash
+# Backup environment file
+cp .env.production .env.production.backup
+
+# Backup docker-compose
+cp docker-compose.prod.yml docker-compose.prod.yml.backup
+```
+
+## Security Hardening
+
+### Production Security Checklist
+
+- [ ] HTTPS enabled (TLS 1.2+)
+- [ ] Security headers configured
+- [ ] Secrets stored securely (not in code)
+- [ ] Database access restricted (firewall)
+- [ ] Redis access restricted (firewall)
+- [ ] Rate limiting enabled
+- [ ] Authentication required
+- [ ] Error messages masked in production
+- [ ] Logging configured (no sensitive data)
+- [ ] Regular security updates
+
+### SSL/TLS Setup
+
+Use a reverse proxy (nginx, Traefik) for SSL termination:
+
+```nginx
+server {
+    listen 443 ssl;
+    server_name api.yourdomain.com;
+    
+    ssl_certificate /path/to/cert.pem;
+    ssl_certificate_key /path/to/key.pem;
+    
+    location / {
+        proxy_pass http://gateway:3000;
+    }
+}
+```
+
+## Troubleshooting
+
+See [TROUBLESHOOTING.md](./TROUBLESHOOTING.md) for common deployment issues.
+
+## Support
+
+For deployment issues:
+- Check logs: `docker-compose logs gateway`
+- Verify health: `curl http://localhost/health`
+- Review metrics: `curl http://localhost/metrics`
+- See troubleshooting guide
+