4runr-os 2.10.85 → 2.10.86

package/apps/gateway/TROUBLESHOOTING.md

@@ -1,541 +0,0 @@
-# Troubleshooting Guide
-
-Common issues and solutions for the 4Runr Gateway.
-
-## Table of Contents
-
-1. [Startup Issues](#startup-issues)
-2. [Database Issues](#database-issues)
-3. [Authentication Issues](#authentication-issues)
-4. [Queue Issues](#queue-issues)
-5. [Rate Limiting Issues](#rate-limiting-issues)
-6. [Performance Issues](#performance-issues)
-7. [Error Messages](#error-messages)
-
-## Startup Issues
-
-### Gateway Won't Start
-
-**Symptoms:**
-- Container exits immediately
-- "Cannot find module" errors
-- Port already in use
-
-**Solutions:**
-
-1. **Check logs:**
-```bash
-docker-compose logs gateway
-# or
-pm2 logs gateway
-```
-
-2. **Port conflict:**
-```bash
-# Check what's using port 3000
-lsof -i :3000
-# or
-netstat -tulpn | grep 3000
-
-# Change port in docker-compose.yml or .env
-PORT=3001
-```
-
-3. **Missing dependencies:**
-```bash
-# Rebuild
-docker-compose build --no-cache gateway
-
-# Or reinstall
-pnpm install
-```
-
-4. **Prisma client not generated:**
-```bash
-cd apps/gateway
-pnpm db:generate
-```
-
-### Health Check Failing
-
-**Symptoms:**
-- `/health` returns 500 or times out
-- Container marked as unhealthy
-
-**Solutions:**
-
-1. **Check if server is actually running:**
-```bash
-curl http://localhost:3000/health
-```
-
-2. **Check dependencies:**
-```bash
-curl http://localhost:3000/ready
-# Check which dependencies are failing
-```
-
-3. **Review startup logs:**
-```bash
-docker-compose logs gateway | grep -i error
-```
-
-## Database Issues
-
-### Cannot Connect to Database
-
-**Symptoms:**
-- "Connection refused" errors
-- "Authentication failed" errors
-- Timeout errors
-
-**Solutions:**
-
-1. **Verify database is running:**
-```bash
-docker-compose ps postgres
-# or
-psql -h localhost -U 4runr -d 4runr_gateway
-```
-
-2. **Check connection string:**
-```bash
-echo $DATABASE_URL
-# Should be: postgresql://user:password@host:port/database
-```
-
-3. **Verify network connectivity:**
-```bash
-# From gateway container
-docker-compose exec gateway ping postgres
-```
-
-4. **Check firewall rules:**
-```bash
-# Ensure port 5432 is accessible
-telnet db-host 5432
-```
-
-### Migration Errors
-
-**Symptoms:**
-- "Migration failed" errors
-- Schema mismatch errors
-
-**Solutions:**
-
-1. **Check migration status:**
-```bash
-cd apps/gateway
-pnpm prisma migrate status
-```
-
-2. **Reset database (development only):**
-```bash
-pnpm prisma migrate reset
-```
-
-3. **Apply pending migrations:**
-```bash
-pnpm db:migrate
-```
-
-4. **Manual migration:**
-```bash
-pnpm prisma migrate dev --name migration-name
-```
-
-### Prisma Client Not Found
-
-**Symptoms:**
-- "Cannot find module '@prisma/client'"
-- "Prisma client not generated"
-
-**Solutions:**
-
-1. **Generate Prisma client:**
-```bash
-cd apps/gateway
-pnpm db:generate
-```
-
-2. **Verify generation:**
-```bash
-ls -la node_modules/.prisma/client
-```
-
-3. **Rebuild:**
-```bash
-pnpm clean
-pnpm build
-```
-
-## Authentication Issues
-
-### API Key Not Working
-
-**Symptoms:**
-- 401 Unauthorized errors
-- "Invalid API key" messages
-
-**Solutions:**
-
-1. **Verify API key format:**
-```bash
-# Check header name (should be x-api-key)
-curl -H "x-api-key: your-key" http://localhost:3000/api/runs
-```
-
-2. **Check if key exists in database:**
-```bash
-# PostgreSQL mode
-psql -h localhost -U 4runr -d 4runr_gateway -c "SELECT * FROM api_keys;"
-
-# Memory mode - check API_KEY env var
-echo $API_KEY
-```
-
-3. **Verify AUTH_ENABLED:**
-```bash
-# Should be true
-echo $AUTH_ENABLED
-```
-
-4. **Check key is not revoked:**
-```bash
-psql -h localhost -U 4runr -d 4runr_gateway -c "SELECT * FROM api_keys WHERE revoked = false;"
-```
-
-### JWT Authentication Failing
-
-**Symptoms:**
-- "Invalid token" errors
-- "Token expired" errors
-
-**Solutions:**
-
-1. **Verify JWT_SECRET matches:**
-```bash
-echo $JWT_SECRET
-# Must match the secret used to sign the token
-```
-
-2. **Check token expiration:**
-```bash
-# Decode JWT (use jwt.io or similar)
-# Verify 'exp' claim is in the future
-```
-
-3. **Verify algorithm:**
-```bash
-# Check JWT_ALG matches token algorithm
-echo $JWT_ALG
-```
-
-4. **Check token format:**
-```bash
-# Should be: Authorization: Bearer <token>
-curl -H "Authorization: Bearer your-token" http://localhost:3000/api/runs
-```
-
-## Queue Issues
-
-### Jobs Not Processing
-
-**Symptoms:**
-- Runs stuck in "queued" status
-- No job activity
-
-**Solutions:**
-
-1. **Check Redis connection:**
-```bash
-# Verify Redis is running
-docker-compose ps redis
-redis-cli ping
-```
-
-2. **Check REDIS_URL:**
-```bash
-echo $REDIS_URL
-# Should be: redis://host:port
-```
-
-3. **Verify queue initialization:**
-```bash
-# Check logs for queue initialization
-docker-compose logs gateway | grep -i queue
-```
-
-4. **Check queue status:**
-```bash
-# Access Redis and check queue
-redis-cli
-> KEYS bull:run-execution:*
-> LLEN bull:run-execution:waiting
-```
-
-### Jobs Failing
-
-**Symptoms:**
-- Jobs marked as "failed"
-- Error logs in processor
-
-**Solutions:**
-
-1. **Check job error logs:**
-```bash
-docker-compose logs gateway | grep -i "job.*fail"
-```
-
-2. **Verify processor context:**
-```bash
-# Check if Sentinel/SSE context is set
-docker-compose logs gateway | grep -i "processor context"
-```
-
-3. **Check run store:**
-```bash
-# Verify runs can be retrieved
-curl http://localhost:3000/api/runs -H "x-api-key: your-key"
-```
-
-## Rate Limiting Issues
-
-### Rate Limits Too Strict
-
-**Symptoms:**
-- 429 errors too frequently
-- Legitimate requests blocked
-
-**Solutions:**
-
-1. **Increase limits:**
-```bash
-RATE_LIMIT_MAX=100
-RATE_LIMIT_WINDOW_MS=60000
-```
-
-2. **Check per-route limits:**
-- Read operations: Full limit
-- Write operations: 75% of limit
-- Expensive operations: 50% of limit
-
-3. **Verify Redis is working:**
-```bash
-# Rate limiting uses Redis
-redis-cli ping
-```
-
-4. **Check identity tracking:**
-```bash
-# Rate limits are per-identity (API key, JWT sub, or IP)
-# Verify same identity isn't being used for multiple clients
-```
-
-### Rate Limits Not Working
-
-**Symptoms:**
-- No 429 errors even with high load
-- Rate limiting disabled
-
-**Solutions:**
-
-1. **Verify RATE_LIMIT_ENABLED:**
-```bash
-echo $RATE_LIMIT_ENABLED
-# Should be true
-```
-
-2. **Check Redis connection:**
-```bash
-# Rate limiting requires Redis
-redis-cli ping
-```
-
-3. **Verify middleware is registered:**
-```bash
-# Check logs for rate limit middleware
-docker-compose logs gateway | grep -i "rate limit"
-```
-
-## Performance Issues
-
-### Slow Response Times
-
-**Symptoms:**
-- High latency
-- Timeout errors
-
-**Solutions:**
-
-1. **Check database performance:**
-```bash
-# Monitor database connections
-psql -c "SELECT count(*) FROM pg_stat_activity;"
-
-# Check slow queries
-psql -c "SELECT * FROM pg_stat_statements ORDER BY total_time DESC LIMIT 10;"
-```
-
-2. **Check queue depth:**
-```bash
-# High queue depth indicates bottleneck
-redis-cli LLEN bull:run-execution:waiting
-```
-
-3. **Monitor resource usage:**
-```bash
-# CPU and memory
-docker stats 4runr-gateway
-# or
-top -p $(pgrep -f "node.*gateway")
-```
-
-4. **Check for memory leaks:**
-```bash
-# Monitor memory over time
-watch -n 1 'docker stats --no-stream 4runr-gateway | grep gateway'
-```
-
-### High Memory Usage
-
-**Symptoms:**
-- Container OOM errors
-- Memory usage growing over time
-
-**Solutions:**
-
-1. **Check for memory leaks:**
-```bash
-# Use Node.js memory profiling
-node --inspect dist/index.js
-```
-
-2. **Review queue job retention:**
-```bash
-# Check queue config for job retention
-# Reduce removeOnComplete/removeOnFail if too many jobs retained
-```
-
-3. **Limit concurrent jobs:**
-```bash
-# Configure queue concurrency
-# In queue config, set concurrency limit
-```
-
-4. **Increase container memory:**
-```bash
-# In docker-compose.yml
-deploy:
-  resources:
-    limits:
-      memory: 2G
-```
-
-## Error Messages
-
-### "Cannot find module '@prisma/client'"
-
-**Solution:**
-```bash
-cd apps/gateway
-pnpm db:generate
-```
-
-### "Prisma client not generated yet"
-
-**Solution:**
-```bash
-cd apps/gateway
-pnpm db:generate
-pnpm build
-```
-
-### "Queue system not available (Redis required)"
-
-**Solution:**
-- Verify Redis is running: `docker-compose ps redis`
-- Check REDIS_URL: `echo $REDIS_URL`
-- Gateway will fall back to synchronous execution
-
-### "Security validation failed"
-
-**Solution:**
-- Check JWT_SECRET length (min 32 chars)
-- Verify no insecure default values
-- See security validation logs for details
-
-### "Database connection required"
-
-**Solution:**
-- Verify DATABASE_URL is set
-- Check database is accessible
-- In development, can use `GATEWAY_PERSISTENCE=memory`
-
-### "Rate limit exceeded"
-
-**Solution:**
-- Wait for rate limit window to reset
-- Check retry_after_ms in error response
-- Consider increasing RATE_LIMIT_MAX if legitimate
-
-## Getting Help
-
-### Collecting Diagnostic Information
-
-```bash
-# Health status
-curl http://localhost:3000/health
-curl http://localhost:3000/ready
-
-# Metrics
-curl http://localhost:3000/metrics
-
-# Logs
-docker-compose logs gateway --tail=100
-
-# Environment
-docker-compose exec gateway env | grep -E "(NODE_ENV|DATABASE|REDIS|AUTH|RATE)"
-
-# Database status
-psql -h localhost -U 4runr -d 4runr_gateway -c "SELECT version();"
-
-# Redis status
-redis-cli INFO
-```
-
-### Log Levels
-
-Increase log verbosity for debugging:
-
-```bash
-LOG_LEVEL=debug
-```
-
-**Warning**: Debug logs can be verbose. Use only for troubleshooting.
-
-## Common Patterns
-
-### Service Won't Start After Update
-
-1. Check for breaking changes in changelog
-2. Verify environment variables still valid
-3. Check database migrations needed
-4. Review dependency updates
-
-### Intermittent Errors
-
-1. Check for resource limits (CPU, memory)
-2. Verify network stability
-3. Check database connection pool
-4. Review queue job processing
-
-### Production Issues
-
-1. Check error logs first
-2. Verify health endpoints
-3. Review metrics for anomalies
-4. Check recent deployments
-5. Verify dependencies (DB, Redis) are healthy
-

package/apps/gateway/docker-compose.local.yml

@@ -1,46 +0,0 @@
-# 4Runr Local Gateway - Docker Compose
-# Lightweight Postgres + Redis for local development
-# Volumes persist data across restarts and npm updates
-
-name: 4runr
-
-services:
-  postgres:
-    image: postgres:15-alpine
-    container_name: 4runr-postgres
-    environment:
-      POSTGRES_DB: 4runr_local
-      POSTGRES_USER: 4runr
-      POSTGRES_PASSWORD: ${DB_PASSWORD:-4runr_dev_pass}
-    ports:
-      - "5432:5432"
-    volumes:
-      - 4runr-postgres-data:/var/lib/postgresql/data
-    healthcheck:
-      # Must use -d: default DB name would match user ("4runr") but POSTGRES_DB is 4runr_local
-      test: ["CMD-SHELL", "pg_isready -U 4runr -d 4runr_local"]
-      interval: 5s
-      timeout: 5s
-      retries: 5
-    restart: unless-stopped
-
-  redis:
-    image: redis:7-alpine
-    container_name: 4runr-redis
-    ports:
-      - "6379:6379"
-    volumes:
-      - 4runr-redis-data:/data
-    command: redis-server --appendonly yes
-    healthcheck:
-      test: ["CMD", "redis-cli", "ping"]
-      interval: 5s
-      timeout: 3s
-      retries: 5
-    restart: unless-stopped
-
-volumes:
-  4runr-postgres-data:
-    name: 4runr-postgres-data
-  4runr-redis-data:
-    name: 4runr-redis-data

package/apps/gateway/find-gateway.sh

@@ -1,37 +0,0 @@
-#!/bin/bash
-# Script to help find where the gateway code is located
-
-echo "Looking for gateway code..."
-echo ""
-
-# Check common locations
-LOCATIONS=(
-    "$HOME/4Runr-AI-Agent-OS"
-    "$HOME/4runr/gateway"
-    "/opt/4runr/gateway"
-    "/opt/4Runr-AI-Agent-OS"
-    "$(pwd)/4Runr-AI-Agent-OS"
-    "$(pwd)"
-)
-
-for loc in "${LOCATIONS[@]}"; do
-    if [ -d "$loc/apps/gateway" ] && [ -f "$loc/apps/gateway/test-all-phases.sh" ]; then
-        echo "✓ Found gateway at: $loc"
-        echo "  Test script: $loc/apps/gateway/test-all-phases.sh"
-        echo ""
-        echo "To run tests:"
-        echo "  cd $loc/apps/gateway"
-        echo "  bash test-all-phases.sh"
-        exit 0
-    fi
-done
-
-echo "Gateway not found in common locations."
-echo ""
-echo "Current directory: $(pwd)"
-echo "Current repo: $(basename $(git rev-parse --show-toplevel 2>/dev/null || echo 'not a git repo'))"
-echo ""
-echo "Please navigate to the 4Runr-AI-Agent-OS repository and run:"
-echo "  cd apps/gateway"
-echo "  bash test-all-phases.sh"
-

package/apps/gateway/jest.config.cjs

@@ -1,52 +0,0 @@
-module.exports = {
-  preset: 'ts-jest',
-  testEnvironment: 'node',
-  roots: ['<rootDir>/src'],
-  testMatch: ['**/__tests__/**/*.test.ts', '**/__tests__/**/*.spec.ts'],
-  transform: {
-    '^.+\\.ts$': ['ts-jest', {
-      tsconfig: {
-        target: 'ES2022',
-        module: 'CommonJS', // Use CommonJS for Jest
-        moduleResolution: 'node',
-        esModuleInterop: true,
-        allowSyntheticDefaultImports: true,
-        strict: false, // Less strict for tests
-        skipLibCheck: true,
-        forceConsistentCasingInFileNames: false, // Allow index signature access
-        resolveJsonModule: true,
-        declaration: false, // Don't need declarations for tests
-        declarationMap: false,
-        sourceMap: true,
-        noImplicitAny: false, // Allow implicit any in tests
-      },
-      useESM: false, // Use CommonJS for Jest even though package.json has "type": "module"
-      isolatedModules: true, // Skip type checking - just transpile
-    }],
-  },
-  moduleFileExtensions: ['ts', 'js', 'json'],
-  moduleNameMapper: {
-    '^@4runr/shared$': '<rootDir>/../../packages/shared/src',
-    '^@4runr/shared/(.*)$': '<rootDir>/../../packages/shared/src/$1',
-    '^@4runr/sentinel$': '<rootDir>/../../packages/sentinel/src',
-    '^@4runr/sentinel/(.*)$': '<rootDir>/../../packages/sentinel/src/$1',
-    '^@4runr/shield$': '<rootDir>/../../packages/shield/src',
-    '^@4runr/shield/(.*)$': '<rootDir>/../../packages/shield/src/$1',
-    '^(\\.{1,2}/.*)\\.js$': '$1',
-    '^(\\.{1,2}/.*)\\.ts$': '$1',
-  },
-  testPathIgnorePatterns: [
-    '/node_modules/',
-    '/dist/',
-  ],
-  collectCoverageFrom: [
-    'src/**/*.ts',
-    '!src/**/*.d.ts',
-    '!src/**/__tests__/**',
-  ],
-  setupFilesAfterEnv: [],
-  testTimeout: 30000,
-  // Ignore dist directory
-  modulePathIgnorePatterns: ['<rootDir>/dist'],
-};
-