4runr-os 2.10.85 → 2.10.86

package/apps/os-server/PHASE5-TESTING.md

@@ -0,0 +1,222 @@
+# Phase 5: Testing & Validation
+
+This document outlines the testing strategy and test suites for the 4Runr Gateway.
+
+## Overview
+
+Phase 5 includes:
+- **5.1 Integration Tests** (1 day) - Test against real services
+- **5.2 Load Testing** (1 day) - Performance and stress testing
+- **5.3 End-to-End Testing** (0.5-1 day) - Complete workflow validation
+
+## 5.1 Integration Tests
+
+### Test Suites
+
+All integration tests are located in `apps/os-server/src/__tests__/integration/`:
+
+1. **PostgreSQL Persistence** (`postgres-persistence.test.ts`)
+   - CRUD operations
+   - Data persistence across restarts
+   - Status updates
+
+2. **Authentication** (`authentication.test.ts`)
+   - API key authentication
+   - JWT authentication
+   - Mixed mode authentication
+   - Public endpoint access
+
+3. **Idempotency** (`idempotency.test.ts`)
+   - Duplicate request handling
+   - Key expiration
+   - Cross-route idempotency
+
+4. **Sentinel Integration** (`sentinel.test.ts`)
+   - Policy enforcement
+   - Kill switch
+   - Event streaming
+   - Audit trails
+
+5. **Rate Limiting** (`rate-limiting.test.ts`)
+   - Per-tenant rate limiting
+   - Rate limit recovery
+   - Bypass for health endpoints
+   - Different limits per route
+
+6. **End-to-End Workflows** (`e2e-workflow.test.ts`)
+   - Complete run lifecycle
+   - Error scenarios
+   - Integration with all features
+
+### Running Integration Tests
+
+```bash
+# Setup test environment
+cd apps/os-server
+bash scripts/setup-test-env.sh
+
+# Run all integration tests
+pnpm test:integration
+
+# Run specific test suite
+pnpm test -- postgres-persistence
+pnpm test -- authentication
+pnpm test -- idempotency
+pnpm test -- sentinel
+pnpm test -- rate-limiting
+pnpm test -- e2e-workflow
+```
+
+### Prerequisites
+
+1. **Server Running**: Gateway must be running (via docker-compose or manually)
+2. **PostgreSQL**: For persistence tests
+3. **Redis**: For idempotency and queue tests
+4. **Environment Variables**:
+   - `TEST_SERVER_URL` (default: http://localhost:3000)
+   - `TEST_API_KEY` (default: test-key-ce67dabb0d4e27e3d72877c921b89cae)
+   - `TEST_DATABASE_URL` (optional, uses DATABASE_URL if not set)
+   - `TEST_REDIS_URL` (optional, uses REDIS_URL if not set)
+
+## 5.2 Load Testing
+
+### k6 Load Tests
+
+Load tests are located in `apps/os-server/load-tests/`:
+
+1. **Basic Load Test** (`k6-basic.js`)
+   - Ramp up: 10 → 50 users
+   - Tests: Health, List, Create, Get
+   - Duration: ~3 minutes
+
+2. **Rate Limit Test** (`k6-rate-limit.js`)
+   - Rapid ramp to 100 users
+   - Tests rate limiting enforcement
+   - Duration: ~50 seconds
+
+### Running Load Tests
+
+```bash
+# Install k6 (if not installed)
+# See load-tests/README.md for installation instructions
+
+# Set environment variables
+export GATEWAY_URL="http://localhost:3000"
+export API_KEY="test-key-ce67dabb0d4e27e3d72877c921b89cae"
+
+# Run basic load test
+cd apps/os-server
+pnpm load-test
+
+# Run rate limit test
+pnpm load-test:rate-limit
+
+# Or directly with k6
+k6 run load-tests/k6-basic.js
+k6 run load-tests/k6-rate-limit.js
+```
+
+### Performance Targets
+
+- **Response Time**: p95 < 500ms, p99 < 1s
+- **Error Rate**: < 1%
+- **Throughput**: Varies by endpoint
+- **Memory**: No leaks, stable usage
+
+### Memory Leak Testing
+
+Monitor memory during load tests:
+
+```bash
+# Run load test and monitor memory
+k6 run load-tests/k6-basic.js &
+PID=$!
+while kill -0 $PID 2>/dev/null; do
+  ps -p $PID -o rss=,vsz=,pcpu=
+  sleep 5
+done
+```
+
+## 5.3 End-to-End Testing
+
+End-to-end tests are included in the integration test suite (`e2e-workflow.test.ts`).
+
+### Test Scenarios
+
+1. **Complete Run Workflow**
+   - Create → Start → Monitor → Complete
+   - Verifies all status transitions
+   - Checks for output and completion time
+
+2. **Workflow with Authentication**
+   - Verifies auth is required
+   - Tests with valid/invalid credentials
+
+3. **Workflow with Sentinel**
+   - Verifies Sentinel monitoring
+   - Tests policy enforcement
+
+4. **Workflow with Queue**
+   - Tests multiple runs in queue
+   - Verifies priority handling
+
+5. **Error Scenarios**
+   - Invalid run ID
+   - Validation errors
+   - Cancellation
+
+### Running E2E Tests
+
+```bash
+# Run all E2E tests
+pnpm test -- e2e-workflow
+
+# Or run all integration tests (includes E2E)
+pnpm test:integration
+```
+
+## Test Helpers
+
+### `helpers/test-server.ts`
+
+Provides utilities for making HTTP requests to the test server:
+- `initTestServer()` - Initialize test configuration
+- `authenticatedRequest()` - Make authenticated requests
+- `jwtAuthenticatedRequest()` - Make JWT-authenticated requests
+- `unauthenticatedRequest()` - Make unauthenticated requests
+- `waitForServer()` - Wait for server to be ready
+
+## Success Criteria
+
+### Integration Tests
+- ✅ All integration tests pass
+- ✅ Edge cases covered
+- ✅ Tests are isolated and clean up after themselves
+
+### Load Testing
+- ✅ System handles expected load (50+ concurrent users)
+- ✅ Response times acceptable (p95 < 500ms)
+- ✅ No memory leaks (stable memory usage)
+- ✅ Rate limiting effective under load
+
+### End-to-End Testing
+- ✅ Complete workflows work end-to-end
+- ✅ All features integrated correctly
+- ✅ Error scenarios handled gracefully
+
+## Continuous Integration
+
+For CI/CD, ensure:
+1. Test services are available (PostgreSQL, Redis)
+2. Gateway server is started before tests
+3. Tests clean up after themselves
+4. Load tests are optional (can be run separately)
+
+## Next Steps
+
+After Phase 5 completion:
+- Review test coverage
+- Add missing edge cases
+- Optimize based on load test results
+- Document performance baselines
+

package/apps/os-server/README.md

@@ -0,0 +1,368 @@
+# 4Runr Gateway
+
+The 4Runr Gateway is a production-ready HTTP server for AI agent orchestration and execution management.
+
+## Quick Start
+
+### Local Development (with Docker)
+
+```bash
+# Prerequisites: Docker Desktop (Windows/macOS) or Docker Engine (Linux)
+
+# 1. Install dependencies
+npm install
+
+# 2. Build Gateway
+cd apps/os-server
+npm run build
+
+# 3. Start Gateway (auto-starts Docker containers)
+npm run dev
+# Or from workspace root:
+npm run gateway:dev
+
+# Gateway starts on http://localhost:3001
+# Postgres on localhost:5432, Redis on localhost:6379
+```
+
+### Development without Docker (Memory Mode)
+
+```bash
+export GATEWAY_PERSISTENCE=memory
+export AUTH_ENABLED=false
+npm run dev
+```
+
+## Environment Variables
+
+### Core Configuration
+
+| Variable | Values | Default | Description |
+|----------|--------|---------|-------------|
+| `NODE_ENV` | `development`, `production` | `development` | Environment mode |
+| `PORT` | `1-65535` | `3001` | HTTP server port |
+| `HOST` | IP address | `127.0.0.1` | Server bind address |
+| `LOG_LEVEL` | `error`, `warn`, `info`, `debug` | `info` | Logging verbosity |
+
+### Persistence & Database
+
+| Variable | Values | Default | Description |
+|----------|--------|---------|-------------|
+| `GATEWAY_PERSISTENCE` | `auto`, `memory`, `file`, `postgres` | `auto` | Persistence mode |
+| `DATABASE_URL` | PostgreSQL connection string | `postgresql://4runr:4runr_dev_pass@localhost:5432/4runr_local` | Database connection |
+| `REDIS_URL` | Redis connection string | `redis://localhost:6379` | Redis connection |
+
+**Persistence Modes:**
+- **`auto`** (recommended): Detects Docker and auto-starts Postgres+Redis
+- **`memory`**: In-memory only, no persistence (fast, ephemeral)
+- **`postgres`**: Explicit PostgreSQL mode (production)
+- **`file`**: Reserved for future SQLite support
+
+### Authentication & Security
+
+| Variable | Values | Default | Description |
+|----------|--------|---------|-------------|
+| `AUTH_ENABLED` | `true`, `false` | `true` | Enable authentication |
+| `AUTH_MODE` | `api_key`, `jwt`, `mixed` | `api_key` | Authentication method |
+| `JWT_SECRET` | String | (required in production) | JWT signing secret |
+| `ENCRYPTION_KEY` | String | (required in production) | Data encryption key |
+| `API_KEYS` | Comma-separated | (none) | Valid API keys |
+
+### Rate Limiting
+
+| Variable | Values | Default | Description |
+|----------|--------|---------|-------------|
+| `RATE_LIMIT_ENABLED` | `true`, `false` | `true` | Enable rate limiting |
+| `RATE_LIMIT_WINDOW_MS` | Number (ms) | `60000` | Rate limit window (1 min) |
+| `RATE_LIMIT_MAX` | Number | `60` | Max requests per window |
+
+### Features
+
+| Variable | Values | Default | Description |
+|----------|--------|---------|-------------|
+| `DEVKIT_ENABLED` | `true`, `false` | `false` | Enable DevKit routes |
+| `METRICS_ENABLED` | `true`, `false` | `true` | Enable Prometheus metrics |
+| `METRICS_PORT` | `1-65535` | `9090` | Metrics endpoint port |
+| `CACHE_ENABLED` | `true`, `false` | `false` | Enable response caching |
+| `IDEMPOTENCY_ENABLED` | `true`, `false` | `false` | Enable idempotency |
+
+### Sentinel (AI Safety)
+
+| Variable | Values | Default | Description |
+|----------|--------|---------|-------------|
+| `SENTINEL_ENABLED` | `true`, `false` | `true` | Enable Sentinel checks |
+| `SENTINEL_MODE` | `live`, `mock` | `live` | Sentinel mode |
+| `SENTINEL_SHIELD_MODE` | `enforce`, `monitor`, `off` | `enforce` | Shield behavior |
+| `SHIELD_TOKEN_CAP` | Number | (optional) | Max tokens per request |
+| `RUN_MAX_TOKENS` | Number | (optional) | Max tokens per run |
+
+### CORS & Security Headers
+
+| Variable | Values | Default | Description |
+|----------|--------|---------|-------------|
+| `CORS_ENABLED` | `true`, `false` | `false` | Enable CORS |
+| `CORS_ORIGIN` | String | `*` | Allowed origins |
+| `ALLOW_LOCALHOST_ONLY` | `true`, `false` | `true` | Restrict to localhost |
+
+## Docker Setup
+
+### Auto-Managed (Recommended)
+
+Gateway automatically manages Docker containers when `GATEWAY_PERSISTENCE=auto`:
+
+**What happens:**
+1. Gateway checks for Docker availability
+2. Looks for existing `4runr-postgres` and `4runr-redis` containers
+3. If not found, runs `docker-compose up -d`
+4. Waits for health checks
+5. Runs database migrations
+6. Connects and starts
+
+**Containers created:**
+- `4runr-postgres`: PostgreSQL 15 Alpine (~25MB)
+- `4runr-redis`: Redis 7 Alpine (~15MB)
+
+**Data persistence:**
+- `4runr-postgres-data`: Docker volume (survives Gateway restarts and npm updates)
+- `4runr-redis-data`: Docker volume (survives Gateway restarts and npm updates)
+
+### Manual Docker Management
+
+```bash
+# Start containers manually (from apps/os-server/)
+docker-compose -f docker-compose.local.yml up -d
+
+# Check status
+docker ps --filter "name=4runr"
+
+# Check health
+docker inspect 4runr-postgres --format='{{.State.Health.Status}}'
+
+# View logs
+docker logs -f 4runr-postgres
+docker logs -f 4runr-redis
+
+# Stop containers (data persists)
+docker-compose -f docker-compose.local.yml down
+
+# Stop and remove data
+docker-compose -f docker-compose.local.yml down -v
+```
+
+### Docker Compose Configuration
+
+Location: `apps/os-server/docker-compose.local.yml`
+
+```yaml
+services:
+  postgres:
+    image: postgres:15-alpine
+    container_name: 4runr-postgres
+    environment:
+      POSTGRES_DB: 4runr_local
+      POSTGRES_USER: 4runr
+      POSTGRES_PASSWORD: ${DB_PASSWORD:-4runr_dev_pass}
+    ports:
+      - "5432:5432"
+    volumes:
+      - 4runr-postgres-data:/var/lib/postgresql/data
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U 4runr -d 4runr_local"]
+      interval: 5s
+      timeout: 5s
+      retries: 5
+
+  redis:
+    image: redis:7-alpine
+    container_name: 4runr-redis
+    ports:
+      - "6379:6379"
+    volumes:
+      - 4runr-redis-data:/data
+    command: redis-server --appendonly yes
+    healthcheck:
+      test: ["CMD", "redis-cli", "ping"]
+      interval: 5s
+      timeout: 3s
+      retries: 5
+```
+
+## Database Migrations
+
+Migrations run automatically on Gateway startup when Docker mode is active.
+
+**Manual migrations:**
+```bash
+# Generate Prisma client
+cd apps/os-server
+npx prisma generate
+
+# Create new migration
+npx prisma migrate dev --name migration_name
+
+# Deploy migrations (production)
+npx prisma migrate deploy
+
+# Reset database (development only, deletes all data)
+npx prisma migrate reset
+```
+
+**Schema location:** `prisma/schema.prisma`
+
+## API Endpoints
+
+### Health & Status
+
+- `GET /health` - Health check (always available)
+- `GET /ready` - Readiness check (checks dependencies)
+- `GET /metrics` - Prometheus metrics (port 9090)
+
+### Runs API
+
+- `POST /api/runs` - Create a new run
+- `POST /api/runs/:id/start` - Start a run
+- `POST /api/runs/:id/cancel` - Cancel a running run
+- `GET /api/runs/:id/status` - Get run status
+- `GET /api/runs/:id/stream` - SSE stream for run events
+
+### DevKit API (requires database)
+
+- `GET /devkit/api/status` - System status
+- `GET /devkit/api/health` - Detailed health checks
+- `GET /devkit/api/config` - Non-sensitive configuration
+- `GET /devkit/api/runs` - List recent runs
+- `GET /devkit/api/runs/:id` - Get run details
+- `GET /devkit/api/metrics/summary` - Parsed metrics summary
+
+**Note:** DevKit routes are automatically disabled in memory mode.
+
+## Troubleshooting
+
+### "Docker not available"
+
+**Symptom:**
+```
+[WARN] ⚠️  Docker not available. Running in memory-only mode.
+```
+
+**Solutions:**
+1. Install Docker Desktop (Windows/macOS) or Docker Engine (Linux)
+2. Start Docker: `docker info` should return server info
+3. Check user permissions (Linux): `sudo usermod -aG docker $USER`
+4. WSL2 (Windows): Enable in Docker Desktop settings
+
+### "Port 5432 already allocated"
+
+**Symptom:**
+```
+Error: Bind for 0.0.0.0:5432 failed: port is already allocated
+```
+
+**Solutions:**
+1. Find conflicting container: `docker ps --filter "publish=5432"`
+2. Stop it: `docker stop CONTAINER_NAME`
+3. Or change 4Runr port in `docker-compose.local.yml`
+
+### "Container did not become healthy"
+
+**Symptom:**
+```
+[ERROR] Container 4runr-postgres did not become healthy within 60000ms
+```
+
+**Solutions:**
+1. Pre-pull images: `docker-compose -f docker-compose.local.yml pull`
+2. Start containers manually first: `docker-compose up -d`
+3. Wait for health: `docker ps` shows "(healthy)" status
+
+### More Help
+
+See [`docs/TROUBLESHOOTING.md`](../../docs/TROUBLESHOOTING.md) for comprehensive troubleshooting guide.
+
+## Development Workflow
+
+### Building
+
+```bash
+# Build Gateway only
+cd apps/os-server
+npm run build
+
+# Build all packages
+cd ../..
+npm run build
+```
+
+### Running
+
+```bash
+# Development mode (auto-reload)
+npm run dev
+
+# Production mode
+NODE_ENV=production npm start
+```
+
+### Testing
+
+```bash
+# Run tests
+npm test
+
+# Run with coverage
+npm run test:coverage
+
+# Run specific test
+npm test -- --grep "idempotency"
+```
+
+### Linting
+
+```bash
+# Lint
+npm run lint
+
+# Lint and fix
+npm run lint:fix
+```
+
+## Production Deployment
+
+See [`docs/GATEWAY-PRODUCTION-READINESS-PLAN.md`](../../docs/GATEWAY-PRODUCTION-READINESS-PLAN.md) for production deployment guide.
+
+**Key production requirements:**
+- Set `NODE_ENV=production`
+- Provide `JWT_SECRET` and `ENCRYPTION_KEY`
+- Use external PostgreSQL and Redis
+- Enable TLS/HTTPS
+- Configure monitoring and observability
+- Set up proper rate limiting
+- Review security checklist
+
+## Project Structure
+
+```
+apps/os-server/
+├── src/
+│   ├── index.ts              # Entry point
+│   ├── db/
+│   │   ├── init.ts           # Database initialization
+│   │   ├── docker-manager.ts # Docker container management
+│   │   └── memory-db.ts      # In-memory fallback
+│   ├── routes/               # API route handlers
+│   ├── middleware/           # Fastify middleware
+│   ├── devkit/               # DevKit routes
+│   └── ai-providers/         # AI provider integrations
+├── prisma/
+│   └── schema.prisma         # Database schema
+├── docker-compose.local.yml  # Local Docker setup
+└── package.json
+```
+
+## Additional Documentation
+
+- **Implementation Plan:** [`docs/4RUNR-DB-IMPLEMENTATION-PLAN.md`](../../docs/4RUNR-DB-IMPLEMENTATION-PLAN.md)
+- **Test Results:** [`docs/PHASE-4-TEST-RESULTS.md`](../../docs/PHASE-4-TEST-RESULTS.md)
+- **Troubleshooting:** [`docs/TROUBLESHOOTING.md`](../../docs/TROUBLESHOOTING.md)
+- **Decisions:** [`docs/decisions.md`](../../docs/decisions.md)
+- **Status:** [`docs/status.md`](../../docs/status.md)