4runr-os 2.9.136 → 2.10.0

package/apps/gateway/README.md

@@ -1,351 +1,368 @@
 # 4Runr Gateway
 
-The 4Runr Gateway is a production-ready API gateway for managing AI agent runs with comprehensive features including authentication, rate limiting, queue management, idempotency, security hardening, and Sentinel AI safety integration.
-
-## Features
-
-### Core Features
-- **Run Management**: Create, start, monitor, and manage AI agent runs
-- **PostgreSQL Persistence**: Persistent storage with Prisma ORM
-- **Memory Mode**: Lightweight mode for testing without database
-- **Queue System**: BullMQ-based job queue with priority support
-- **Real-time Updates**: Server-Sent Events (SSE) for run logs and status
-
-### Security & Authentication
-- **Multiple Auth Modes**: API key, JWT, or mixed authentication
-- **Security Headers**: CSP, HSTS, X-Frame-Options, and more
-- **Secrets Validation**: Automatic validation of security configuration
-- **Rate Limiting**: Per-tenant rate limiting with Redis backing
-- **Idempotency**: Prevent duplicate requests with Redis-backed idempotency keys
-
-### Observability
-- **Structured Logging**: Centralized logging with correlation IDs
-- **Prometheus Metrics**: Comprehensive metrics for monitoring
-- **Health Checks**: Liveness, readiness, and startup probes
-- **Error Handling**: Global error handler with production masking
-
-### AI Safety
-- **Sentinel Integration**: AI agent safety monitoring and enforcement
-- **Policy Enforcement**: Max duration, token limits, and custom policies
-- **Kill Switch**: Emergency run termination
-- **Audit Trails**: Complete audit logging
+The 4Runr Gateway is a production-ready HTTP server for AI agent orchestration and execution management.
 
 ## Quick Start
 
-### Prerequisites
-- Node.js >= 18.0.0
-- Docker and Docker Compose (for containerized deployment)
-- PostgreSQL (for persistent mode)
-- Redis (for queue, rate limiting, and idempotency)
+### Local Development (with Docker)
 
-### Local Development
-
-1. **Clone and install dependencies:**
 ```bash
-git clone <repository>
-cd 4Runr-AI-Agent-OS
-pnpm install
-```
+# Prerequisites: Docker Desktop (Windows/macOS) or Docker Engine (Linux)
 
-2. **Set up environment:**
-```bash
+# 1. Install dependencies
+npm install
+
+# 2. Build Gateway
 cd apps/gateway
-cp ../../infra/env.example ../../infra/.env
-# Edit infra/.env with your configuration
-```
+npm run build
 
-3. **Start services:**
-```bash
-cd ../../infra
-docker-compose up -d postgres redis
-```
+# 3. Start Gateway (auto-starts Docker containers)
+npm run dev
+# Or from workspace root:
+npm run gateway:dev
 
-4. **Generate Prisma client:**
-```bash
-cd ../apps/gateway
-pnpm db:generate
+# Gateway starts on http://localhost:3001
+# Postgres on localhost:5432, Redis on localhost:6379
 ```
 
-5. **Run migrations:**
-```bash
-pnpm db:migrate
-```
+### Development without Docker (Memory Mode)
 
-6. **Start the gateway:**
 ```bash
-pnpm dev
+export GATEWAY_PERSISTENCE=memory
+export AUTH_ENABLED=false
+npm run dev
 ```
 
-The gateway will be available at `http://localhost:3000`
+## Environment Variables
 
-### Docker Deployment
+### Core Configuration
 
-```bash
-cd infra
-docker-compose up -d
-```
+| Variable | Values | Default | Description |
+|----------|--------|---------|-------------|
+| `NODE_ENV` | `development`, `production` | `development` | Environment mode |
+| `PORT` | `1-65535` | `3001` | HTTP server port |
+| `HOST` | IP address | `127.0.0.1` | Server bind address |
+| `LOG_LEVEL` | `error`, `warn`, `info`, `debug` | `info` | Logging verbosity |
 
-## Configuration
+### Persistence & Database
 
-### Environment Variables
+| Variable | Values | Default | Description |
+|----------|--------|---------|-------------|
+| `GATEWAY_PERSISTENCE` | `auto`, `memory`, `file`, `postgres` | `auto` | Persistence mode |
+| `DATABASE_URL` | PostgreSQL connection string | `postgresql://4runr:4runr_dev_pass@localhost:5432/4runr_local` | Database connection |
+| `REDIS_URL` | Redis connection string | `redis://localhost:6379` | Redis connection |
 
-See [ENV_VARIABLES.md](./ENV_VARIABLES.md) for complete documentation.
+**Persistence Modes:**
+- **`auto`** (recommended): Detects Docker and auto-starts Postgres+Redis
+- **`memory`**: In-memory only, no persistence (fast, ephemeral)
+- **`postgres`**: Explicit PostgreSQL mode (production)
+- **`file`**: Reserved for future SQLite support
 
-**Quick Configuration:**
-```bash
-# Minimal development setup
-NODE_ENV=development
-GATEWAY_PERSISTENCE=postgres
-DATABASE_URL=postgresql://user:password@localhost:5432/4runr_gateway
-REDIS_URL=redis://localhost:6379
-AUTH_ENABLED=true
-AUTH_MODE=api_key
-JWT_SECRET=your-secret-min-32-chars
-```
+### Authentication & Security
 
-### Persistence Modes
+| Variable | Values | Default | Description |
+|----------|--------|---------|-------------|
+| `AUTH_ENABLED` | `true`, `false` | `true` | Enable authentication |
+| `AUTH_MODE` | `api_key`, `jwt`, `mixed` | `api_key` | Authentication method |
+| `JWT_SECRET` | String | (required in production) | JWT signing secret |
+| `ENCRYPTION_KEY` | String | (required in production) | Data encryption key |
+| `API_KEYS` | Comma-separated | (none) | Valid API keys |
 
-- **`memory`**: No database/filesystem writes (testing only)
-- **`postgres`** or **`persistent`**: PostgreSQL database (production)
+### Rate Limiting
 
-### Authentication Modes
+| Variable | Values | Default | Description |
+|----------|--------|---------|-------------|
+| `RATE_LIMIT_ENABLED` | `true`, `false` | `true` | Enable rate limiting |
+| `RATE_LIMIT_WINDOW_MS` | Number (ms) | `60000` | Rate limit window (1 min) |
+| `RATE_LIMIT_MAX` | Number | `60` | Max requests per window |
 
-- **`api_key`**: API key authentication only
-- **`jwt`**: JWT token authentication only
-- **`mixed`**: Accept either API key or JWT
+### Features
 
-## API Endpoints
+| Variable | Values | Default | Description |
+|----------|--------|---------|-------------|
+| `DEVKIT_ENABLED` | `true`, `false` | `false` | Enable DevKit routes |
+| `METRICS_ENABLED` | `true`, `false` | `true` | Enable Prometheus metrics |
+| `METRICS_PORT` | `1-65535` | `9090` | Metrics endpoint port |
+| `CACHE_ENABLED` | `true`, `false` | `false` | Enable response caching |
+| `IDEMPOTENCY_ENABLED` | `true`, `false` | `false` | Enable idempotency |
 
-### Run Management
+### Sentinel (AI Safety)
 
-- `POST /api/runs` - Create a new run
-- `GET /api/runs` - List runs (with filtering)
-- `GET /api/runs/:id` - Get run details
-- `POST /api/runs/:id/start` - Start a run (queues if queue system available)
-- `POST /api/runs/:id/cancel` - Cancel a run
-- `GET /api/runs/:id/logs/stream` - SSE stream for run logs
+| Variable | Values | Default | Description |
+|----------|--------|---------|-------------|
+| `SENTINEL_ENABLED` | `true`, `false` | `true` | Enable Sentinel checks |
+| `SENTINEL_MODE` | `live`, `mock` | `live` | Sentinel mode |
+| `SENTINEL_SHIELD_MODE` | `enforce`, `monitor`, `off` | `enforce` | Shield behavior |
+| `SHIELD_TOKEN_CAP` | Number | (optional) | Max tokens per request |
+| `RUN_MAX_TOKENS` | Number | (optional) | Max tokens per run |
 
-### Health & Monitoring
+### CORS & Security Headers
 
-- `GET /health` - Liveness probe
-- `GET /ready` - Readiness probe (checks dependencies)
-- `GET /startup` - Startup probe
-- `GET /metrics` - Prometheus metrics
+| Variable | Values | Default | Description |
+|----------|--------|---------|-------------|
+| `CORS_ENABLED` | `true`, `false` | `false` | Enable CORS |
+| `CORS_ORIGIN` | String | `*` | Allowed origins |
+| `ALLOW_LOCALHOST_ONLY` | `true`, `false` | `true` | Restrict to localhost |
 
-### Authentication
+## Docker Setup
 
-All `/api/*` endpoints require authentication (unless `AUTH_ENABLED=false`).
+### Auto-Managed (Recommended)
 
-**API Key:**
-```bash
-curl -H "x-api-key: your-api-key" http://localhost:3000/api/runs
-```
+Gateway automatically manages Docker containers when `GATEWAY_PERSISTENCE=auto`:
 
-**JWT:**
-```bash
-curl -H "Authorization: Bearer your-jwt-token" http://localhost:3000/api/runs
-```
+**What happens:**
+1. Gateway checks for Docker availability
+2. Looks for existing `4runr-postgres` and `4runr-redis` containers
+3. If not found, runs `docker-compose up -d`
+4. Waits for health checks
+5. Runs database migrations
+6. Connects and starts
 
-## Queue System
+**Containers created:**
+- `4runr-postgres`: PostgreSQL 15 Alpine (~25MB)
+- `4runr-redis`: Redis 7 Alpine (~15MB)
 
-The gateway includes a BullMQ-based queue system for asynchronous run execution:
+**Data persistence:**
+- `4runr-postgres-data`: Docker volume (survives Gateway restarts and npm updates)
+- `4runr-redis-data`: Docker volume (survives Gateway restarts and npm updates)
 
-- **Priority Support**: High, normal, and low priority queues
-- **Retry Logic**: Automatic retries with exponential backoff
-- **Job Status**: Track job state (waiting, active, completed, failed)
-- **Graceful Shutdown**: Jobs are properly handled during shutdown
+### Manual Docker Management
 
-Queue is automatically enabled when Redis is available. Falls back to synchronous execution if Redis is unavailable.
+```bash
+# Start containers manually (from apps/gateway/)
+docker-compose -f docker-compose.local.yml up -d
+
+# Check status
+docker ps --filter "name=4runr"
 
-## Rate Limiting
+# Check health
+docker inspect 4runr-postgres --format='{{.State.Health.Status}}'
 
-Rate limiting is enforced per identity (API key, JWT sub, or IP address):
+# View logs
+docker logs -f 4runr-postgres
+docker logs -f 4runr-redis
 
-- **Read Operations**: Full limit (default: 60/min)
-- **Write Operations**: 75% of limit (default: 45/min)
-- **Expensive Operations**: 50% of limit (default: 30/min)
+# Stop containers (data persists)
+docker-compose -f docker-compose.local.yml down
+
+# Stop and remove data
+docker-compose -f docker-compose.local.yml down -v
+```
 
-Rate limit errors include retry information:
-```json
-{
-  "error": "rate_limited",
-  "retry_after_ms": 45000,
-  "limit": 60,
-  "window_ms": 60000
-}
+### Docker Compose Configuration
+
+Location: `apps/gateway/docker-compose.local.yml`
+
+```yaml
+services:
+  postgres:
+    image: postgres:15-alpine
+    container_name: 4runr-postgres
+    environment:
+      POSTGRES_DB: 4runr_local
+      POSTGRES_USER: 4runr
+      POSTGRES_PASSWORD: ${DB_PASSWORD:-4runr_dev_pass}
+    ports:
+      - "5432:5432"
+    volumes:
+      - 4runr-postgres-data:/var/lib/postgresql/data
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U 4runr"]
+      interval: 5s
+      timeout: 5s
+      retries: 5
+
+  redis:
+    image: redis:7-alpine
+    container_name: 4runr-redis
+    ports:
+      - "6379:6379"
+    volumes:
+      - 4runr-redis-data:/data
+    command: redis-server --appendonly yes
+    healthcheck:
+      test: ["CMD", "redis-cli", "ping"]
+      interval: 5s
+      timeout: 3s
+      retries: 5
 ```
 
-## Idempotency
+## Database Migrations
 
-Prevent duplicate requests using idempotency keys:
+Migrations run automatically on Gateway startup when Docker mode is active.
 
+**Manual migrations:**
 ```bash
-curl -X POST http://localhost:3000/api/runs \
-  -H "x-api-key: your-key" \
-  -H "Idempotency-Key: unique-key-123" \
-  -H "Content-Type: application/json" \
-  -d '{"name": "My Run", "input": {}}'
-```
+# Generate Prisma client
+cd apps/gateway
+npx prisma generate
 
-Subsequent requests with the same idempotency key return the cached response.
+# Create new migration
+npx prisma migrate dev --name migration_name
 
-## Security Features
+# Deploy migrations (production)
+npx prisma migrate deploy
 
-### Security Headers
-- Content Security Policy (CSP)
-- HTTP Strict Transport Security (HSTS)
-- X-Content-Type-Options
-- X-Frame-Options
-- X-XSS-Protection
-- Referrer-Policy
-- Permissions-Policy
-- Cross-Origin policies (COEP, COOP, CORP)
+# Reset database (development only, deletes all data)
+npx prisma migrate reset
+```
 
-### Error Handling
-- Production error masking (500 errors hidden from clients)
-- Correlation IDs for error tracking
-- Consistent error response format
-- Validation error details
+**Schema location:** `prisma/schema.prisma`
 
-## Testing
+## API Endpoints
 
-### Unit Tests
-```bash
-pnpm test
-```
+### Health & Status
 
-### Integration Tests
-```bash
-# Start services first
-bash scripts/setup-test-env.sh
+- `GET /health` - Health check (always available)
+- `GET /ready` - Readiness check (checks dependencies)
+- `GET /metrics` - Prometheus metrics (port 9090)
 
-# Run integration tests
-pnpm test:integration
-```
+### Runs API
 
-### Load Testing
-```bash
-# Install k6 first: https://k6.io/docs/getting-started/installation/
-export GATEWAY_URL="http://localhost:3000"
-export API_KEY="your-api-key"
-pnpm load-test
-```
+- `POST /api/runs` - Create a new run
+- `POST /api/runs/:id/start` - Start a run
+- `POST /api/runs/:id/cancel` - Cancel a running run
+- `GET /api/runs/:id/status` - Get run status
+- `GET /api/runs/:id/stream` - SSE stream for run events
 
-See [PHASE5-TESTING.md](./PHASE5-TESTING.md) for detailed testing documentation.
+### DevKit API (requires database)
 
-## Deployment
+- `GET /devkit/api/status` - System status
+- `GET /devkit/api/health` - Detailed health checks
+- `GET /devkit/api/config` - Non-sensitive configuration
+- `GET /devkit/api/runs` - List recent runs
+- `GET /devkit/api/runs/:id` - Get run details
+- `GET /devkit/api/metrics/summary` - Parsed metrics summary
 
-See [DEPLOYMENT.md](./DEPLOYMENT.md) for comprehensive deployment guide.
+**Note:** DevKit routes are automatically disabled in memory mode.
 
-### Quick Production Deployment
+## Troubleshooting
 
-1. **Set up environment variables:**
-```bash
-cp infra/env.example infra/.env.production
-# Edit with production values
-```
+### "Docker not available"
 
-2. **Build and deploy:**
-```bash
-docker-compose -f infra/docker-compose.prod.yml up -d
+**Symptom:**
 ```
-
-3. **Verify deployment:**
-```bash
-curl http://your-server/health
-curl http://your-server/ready
+[WARN] ⚠️  Docker not available. Running in memory-only mode.
 ```
 
-## Monitoring
+**Solutions:**
+1. Install Docker Desktop (Windows/macOS) or Docker Engine (Linux)
+2. Start Docker: `docker info` should return server info
+3. Check user permissions (Linux): `sudo usermod -aG docker $USER`
+4. WSL2 (Windows): Enable in Docker Desktop settings
+
+### "Port 5432 already allocated"
+
+**Symptom:**
+```
+Error: Bind for 0.0.0.0:5432 failed: port is already allocated
+```
 
-### Metrics
+**Solutions:**
+1. Find conflicting container: `docker ps --filter "publish=5432"`
+2. Stop it: `docker stop CONTAINER_NAME`
+3. Or change 4Runr port in `docker-compose.local.yml`
 
-Prometheus metrics are available at `/metrics`:
+### "Container did not become healthy"
 
-- HTTP request metrics (duration, total, errors)
-- Run metrics (created, started, completed, duration)
-- SSE connection metrics
-- Idempotency metrics
-- Rate limiting metrics
-- Authentication metrics
-- Queue job metrics
+**Symptom:**
+```
+[ERROR] Container 4runr-postgres did not become healthy within 60000ms
+```
 
-### Health Checks
+**Solutions:**
+1. Pre-pull images: `docker-compose -f docker-compose.local.yml pull`
+2. Start containers manually first: `docker-compose up -d`
+3. Wait for health: `docker ps` shows "(healthy)" status
 
-- **Liveness** (`/health`): Server is running
-- **Readiness** (`/ready`): Server and dependencies are ready
-- **Startup** (`/startup`): Server has completed startup
+### More Help
 
-### Logging
+See [`docs/TROUBLESHOOTING.md`](../../docs/TROUBLESHOOTING.md) for comprehensive troubleshooting guide.
 
-Structured JSON logging with correlation IDs. Log levels:
-- `error`: Errors only
-- `warn`: Warnings and errors
-- `info`: Info, warnings, and errors (default)
-- `debug`: All logs
+## Development Workflow
 
-## Troubleshooting
+### Building
 
-See [TROUBLESHOOTING.md](./TROUBLESHOOTING.md) for common issues and solutions.
+```bash
+# Build Gateway only
+cd apps/gateway
+npm run build
 
-## Architecture
+# Build all packages
+cd ../..
+npm run build
+```
 
-### Components
+### Running
 
-- **Fastify Server**: High-performance web framework
-- **Prisma ORM**: Database access layer
-- **BullMQ**: Job queue system
-- **Redis**: Caching, rate limiting, idempotency
-- **Sentinel**: AI safety monitoring
-- **Prometheus**: Metrics collection
+```bash
+# Development mode (auto-reload)
+npm run dev
 
-### Data Flow
+# Production mode
+NODE_ENV=production npm start
+```
 
-1. Request → Authentication → Rate Limiting
-2. Idempotency Check → Route Handler
-3. Run Creation → Queue (if available) → Processor
-4. Status Updates → SSE Events → Client
-5. Completion → Metrics Update → Logging
+### Testing
 
-## Development
+```bash
+# Run tests
+npm test
 
-### Project Structure
+# Run with coverage
+npm run test:coverage
 
+# Run specific test
+npm test -- --grep "idempotency"
 ```
-apps/gateway/
-├── src/
-│   ├── __tests__/          # Test files
-│   ├── adapters/           # External service adapters
-│   ├── config/             # Configuration
-│   ├── db/                 # Database clients
-│   ├── health/             # Health check logic
-│   ├── metrics/            # Prometheus metrics
-│   ├── middleware/         # Express/Fastify middleware
-│   ├── queue/              # Queue system
-│   ├── runs/               # Run store implementations
-│   ├── schemas/            # Zod validation schemas
-│   └── index.ts            # Main entry point
-├── scripts/                # Utility scripts
-├── load-tests/             # Load testing scripts
-└── Dockerfile              # Container definition
+
+### Linting
+
+```bash
+# Lint
+npm run lint
+
+# Lint and fix
+npm run lint:fix
 ```
 
-### Adding New Features
+## Production Deployment
 
-1. Create feature branch
-2. Implement feature with tests
-3. Update documentation
-4. Run tests and linting
-5. Submit PR
+See [`docs/GATEWAY-PRODUCTION-READINESS-PLAN.md`](../../docs/GATEWAY-PRODUCTION-READINESS-PLAN.md) for production deployment guide.
 
-## License
+**Key production requirements:**
+- Set `NODE_ENV=production`
+- Provide `JWT_SECRET` and `ENCRYPTION_KEY`
+- Use external PostgreSQL and Redis
+- Enable TLS/HTTPS
+- Configure monitoring and observability
+- Set up proper rate limiting
+- Review security checklist
 
-[Your License Here]
+## Project Structure
 
-## Support
+```
+apps/gateway/
+├── src/
+│   ├── index.ts              # Entry point
+│   ├── db/
+│   │   ├── init.ts           # Database initialization
+│   │   ├── docker-manager.ts # Docker container management
+│   │   └── memory-db.ts      # In-memory fallback
+│   ├── routes/               # API route handlers
+│   ├── middleware/           # Fastify middleware
+│   ├── devkit/               # DevKit routes
+│   └── ai-providers/         # AI provider integrations
+├── prisma/
+│   └── schema.prisma         # Database schema
+├── docker-compose.local.yml  # Local Docker setup
+└── package.json
+```
 
-For issues and questions:
-- GitHub Issues: [repository-url]/issues
-- Documentation: See `docs/` directory
-- Troubleshooting: See [TROUBLESHOOTING.md](./TROUBLESHOOTING.md)
+## Additional Documentation
 
+- **Implementation Plan:** [`docs/4RUNR-DB-IMPLEMENTATION-PLAN.md`](../../docs/4RUNR-DB-IMPLEMENTATION-PLAN.md)
+- **Test Results:** [`docs/PHASE-4-TEST-RESULTS.md`](../../docs/PHASE-4-TEST-RESULTS.md)
+- **Troubleshooting:** [`docs/TROUBLESHOOTING.md`](../../docs/TROUBLESHOOTING.md)
+- **Decisions:** [`docs/decisions.md`](../../docs/decisions.md)
+- **Status:** [`docs/status.md`](../../docs/status.md)

package/apps/gateway/dist/apps/gateway/src/db/docker-manager.d.ts

@@ -0,0 +1,16 @@
+export declare function isDockerAvailable(): Promise<boolean>;
+export declare function isDockerRunning(): Promise<boolean>;
+export declare function areContainersRunning(): Promise<boolean>;
+export declare function startDockerCompose(composeFilePath: string): Promise<void>;
+export declare function stopDockerCompose(composeFilePath: string): Promise<void>;
+export declare function waitForHealthy(containerName: string, timeoutMs?: number): Promise<void>;
+export declare function isPortInUse(port: number): Promise<boolean>;
+export declare function getContainerLogs(containerName: string, lines?: number): string;
+export interface DockerStatus {
+    available: boolean;
+    running: boolean;
+    containersRunning: boolean;
+    error?: string;
+}
+export declare function getDockerStatus(): Promise<DockerStatus>;
+//# sourceMappingURL=docker-manager.d.ts.map

package/apps/gateway/dist/apps/gateway/src/db/docker-manager.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"docker-manager.d.ts","sourceRoot":"","sources":["../../../../../src/db/docker-manager.ts"],"names":[],"mappings":"AAMA,wBAAsB,iBAAiB,IAAI,OAAO,CAAC,OAAO,CAAC,CAW1D;AAKD,wBAAsB,eAAe,IAAI,OAAO,CAAC,OAAO,CAAC,CAWxD;AAKD,wBAAsB,oBAAoB,IAAI,OAAO,CAAC,OAAO,CAAC,CAW7D;AAMD,wBAAsB,kBAAkB,CAAC,eAAe,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC,CAK/E;AAMD,wBAAsB,iBAAiB,CAAC,eAAe,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC,CAK9E;AAOD,wBAAsB,cAAc,CAAC,aAAa,EAAE,MAAM,EAAE,SAAS,GAAE,MAAc,GAAG,OAAO,CAAC,IAAI,CAAC,CAuCpG;AAMD,wBAAsB,WAAW,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,OAAO,CAAC,CAchE;AAOD,wBAAgB,gBAAgB,CAAC,aAAa,EAAE,MAAM,EAAE,KAAK,GAAE,MAAW,GAAG,MAAM,CAUlF;AAYD,MAAM,WAAW,YAAY;IAC3B,SAAS,EAAE,OAAO,CAAC;IACnB,OAAO,EAAE,OAAO,CAAC;IACjB,iBAAiB,EAAE,OAAO,CAAC;IAC3B,KAAK,CAAC,EAAE,MAAM,CAAC;CAChB;AAKD,wBAAsB,eAAe,IAAI,OAAO,CAAC,YAAY,CAAC,CA0B7D"}