4runr-os 2.10.84 → 2.10.86

package/apps/os-server/JWT-RATE-LIMIT-VALIDATION.md

@@ -0,0 +1,462 @@
+# JWT Authentication, Rate Limiting & Schema Validation
+
+## Summary
+
+This document describes the JWT authentication, rate limiting, and schema validation features added to the 4Runr Gateway.
+
+## Features Implemented
+
+### 1. JWT Authentication Layer
+
+**New Files:**
+- `apps/os-server/src/middleware/authJwt.ts` - JWT validation middleware
+- `apps/os-server/src/middleware/auth.ts` - Auth orchestrator for multiple auth modes
+
+**Key Features:**
+- Full JWT token validation with HS256/HS384/HS512 support
+- Expiration checking
+- Signature verification
+- Payload extraction and attachment to request context
+
+**Configuration:**
+```bash
+JWT_SECRET=your-secret-at-least-32-characters-long
+JWT_ALG=HS256
+AUTH_MODE=jwt  # or mixed
+```
+
+**JWT Payload Structure:**
+```typescript
+interface AuthContext {
+  sub?: string;           // Subject (user ID)
+  tenantId?: string;      // Tenant/organization ID
+  roles?: string[];       // User roles
+  exp?: number;           // Expiration time
+  iat?: number;           // Issued at
+  [key: string]: any;     // Additional custom claims
+}
+```
+
+**Usage:**
+```bash
+curl -H "Authorization: Bearer <your-jwt-token>" \
+     http://localhost:3000/api/runs
+```
+
+### 2. Authentication Modes
+
+The gateway now supports three authentication modes:
+
+#### Mode 1: `api_key` (Default)
+Only API key authentication via `x-api-key` header.
+
+```bash
+AUTH_MODE=api_key
+API_KEY=test-key
+
+# Usage
+curl -H "x-api-key: test-key" http://localhost:3000/api/runs
+```
+
+#### Mode 2: `jwt`
+Only JWT authentication via `Authorization: Bearer <token>` header.
+
+```bash
+AUTH_MODE=jwt
+JWT_SECRET=your-secret-key
+
+# Usage
+curl -H "Authorization: Bearer <token>" http://localhost:3000/api/runs
+```
+
+#### Mode 3: `mixed`
+Accept either API key OR JWT. Useful for:
+- Migration scenarios
+- Supporting multiple client types
+- Gradual JWT adoption
+
+```bash
+AUTH_MODE=mixed
+API_KEY=test-key
+JWT_SECRET=your-secret-key
+
+# Usage (either works)
+curl -H "x-api-key: test-key" http://localhost:3000/api/runs
+curl -H "Authorization: Bearer <token>" http://localhost:3000/api/runs
+```
+
+### 3. Rate Limiting
+
+**New Files:**
+- `apps/os-server/src/middleware/rateLimit.ts` - Rate limiting middleware
+
+**Key Features:**
+- Identity-based rate limiting (API key, JWT sub, or IP)
+- Different limits for read/write/expensive operations
+- Configurable window and max requests
+- Machine-consumable error responses
+- Bypass for health/metrics endpoints
+
+**Configuration:**
+```bash
+RATE_LIMIT_ENABLED=true
+RATE_LIMIT_MAX=60              # Max requests per window
+RATE_LIMIT_WINDOW_MS=60000     # Window in ms (60000 = 1 minute)
+```
+
+**Rate Limit Tiers:**
+| Operation Type | Limit |
+|----------------|-------|
+| Read operations | 100% (60/min) |
+| Write operations | 75% (45/min) |
+| Expensive operations | 50% (30/min) |
+
+**Error Response (429):**
+```json
+{
+  "error": "rate_limited",
+  "message": "Too many requests, please try again later",
+  "retry_after_ms": 45000,
+  "limit": 60,
+  "window_ms": 60000
+}
+```
+
+**Identity Priority:**
+1. API key (`x-api-key` header)
+2. JWT subject (`sub` claim)
+3. JWT tenant ID (`tenantId` claim)
+4. IP address (fallback)
+
+### 4. Schema Validation
+
+**New Files:**
+- `apps/os-server/src/middleware/validate.ts` - Validation middleware
+- `apps/os-server/src/schemas/runs.ts` - Schema definitions
+
+**Key Features:**
+- Runtime type validation using Zod
+- Automatic data transformation (e.g., string → number)
+- Detailed error messages with field-level feedback
+- Request body, query params, route params, and header validation
+
+**Validation Helpers:**
+- `validateBody(schema)` - Validate request body
+- `validateQuery(schema)` - Validate query parameters
+- `validateParams(schema)` - Validate route parameters
+- `validateHeaders(schema)` - Validate headers
+
+**Example Schema:**
+```typescript
+const createRunSchema = z.object({
+  name: z.string().min(1).max(200),
+  input: z.unknown().optional(),
+  clientToken: z.string().min(1).max(200).optional(),
+  tags: z.array(z.string().min(1).max(50)).max(20).optional(),
+});
+```
+
+**Validation Error Response (400):**
+```json
+{
+  "error": "invalid_request",
+  "message": "Request body validation failed",
+  "details": {
+    "fieldErrors": {
+      "name": ["Name is required"],
+      "email": ["Invalid email format"]
+    },
+    "formErrors": []
+  }
+}
+```
+
+## Updated Routes
+
+All protected routes now use:
+1. **Auth orchestrator** (`requireAuth`) instead of `requireApiKey`
+2. **Rate limiting** (read/write/strict tiers)
+3. **Schema validation** for inputs
+
+**Example:**
+```typescript
+fastify.post('/api/runs', { 
+  preHandler: [requireAuth, validateBody(createRunSchema)],
+  ...writeRateLimit
+}, async (request, reply) => {
+  // Handler code
+});
+```
+
+**Updated Endpoints:**
+- `POST /api/workspace/plan` - Write rate limit + validation
+- `POST /api/runs` - Write rate limit + validation
+- `GET /api/runs` - Read rate limit + validation
+- `GET /api/runs/:id` - Read rate limit + validation
+- `POST /api/runs/:id/start` - Write rate limit + validation
+- `GET /api/runs/:id/logs/stream` - Read rate limit + validation
+- `POST /api/registry/publish` - Write rate limit + validation
+- `POST /api/privacy/toggle` - Write rate limit + validation
+
+**Unchanged Endpoints (no auth/limits):**
+- `GET /health`
+- `GET /ready`
+- `GET /metrics`
+- `GET /debug/*` (in memory mode)
+
+## Environment Variables
+
+### New Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `AUTH_MODE` | `api_key` | `api_key`, `jwt`, or `mixed` |
+| `JWT_SECRET` | - | JWT signing secret (min 32 chars) |
+| `JWT_ALG` | `HS256` | `HS256`, `HS384`, or `HS512` |
+| `RATE_LIMIT_ENABLED` | `true` | Enable/disable rate limiting |
+| `RATE_LIMIT_MAX` | `60` | Max requests per window |
+| `RATE_LIMIT_WINDOW_MS` | `60000` | Time window in milliseconds |
+
+See `apps/os-server/ENV_VARIABLES.md` for complete documentation.
+
+## Testing
+
+### Unit Tests
+
+**Test Files:**
+- `apps/os-server/src/__tests__/auth.test.ts` - JWT & auth orchestrator tests
+- `apps/os-server/src/__tests__/validation.test.ts` - Schema validation tests
+- `apps/os-server/src/__tests__/rateLimit.test.ts` - Rate limiting tests
+
+**Run Tests:**
+```bash
+cd apps/os-server
+pnpm test
+```
+
+### Manual Testing
+
+#### Test JWT Authentication
+
+1. **Generate a JWT token** (using your preferred tool or online JWT generator):
+   - Payload: `{"sub":"user-123","exp":9999999999}`
+   - Secret: Your `JWT_SECRET`
+   - Algorithm: `HS256`
+
+2. **Test with valid JWT:**
+```bash
+AUTH_MODE=jwt
+curl -H "Authorization: Bearer <your-token>" \
+     http://localhost:3000/api/runs
+```
+
+3. **Test with invalid/expired JWT:**
+```bash
+curl -H "Authorization: Bearer invalid-token" \
+     http://localhost:3000/api/runs
+# Expected: 401 Unauthorized
+```
+
+#### Test Rate Limiting
+
+1. **Send requests rapidly:**
+```bash
+for i in {1..65}; do
+  curl -H "x-api-key: test-key" http://localhost:3000/api/runs
+done
+```
+
+2. **Expected:** First 60 succeed, then 429 errors
+
+3. **Verify error response:**
+```json
+{
+  "error": "rate_limited",
+  "retry_after_ms": 45000,
+  ...
+}
+```
+
+#### Test Validation
+
+1. **Send invalid request:**
+```bash
+curl -X POST http://localhost:3000/api/runs \
+  -H "x-api-key: test-key" \
+  -H "Content-Type: application/json" \
+  -d '{"name":"","input":"invalid"}'
+```
+
+2. **Expected 400 error with field details:**
+```json
+{
+  "error": "invalid_request",
+  "details": {
+    "fieldErrors": {
+      "name": ["String must contain at least 1 character(s)"]
+    }
+  }
+}
+```
+
+3. **Send valid request:**
+```bash
+curl -X POST http://localhost:3000/api/runs \
+  -H "x-api-key: test-key" \
+  -H "Content-Type: application/json" \
+  -d '{"name":"Test Run","input":{"foo":"bar"}}'
+# Expected: 201 Created
+```
+
+## Health Check
+
+The `/health` endpoint now reports auth and rate limiting configuration:
+
+```bash
+curl http://localhost:3000/health
+```
+
+**Response:**
+```json
+{
+  "ok": true,
+  "version": "1.0.0",
+  "time": "2025-11-16T...",
+  "persistence": "postgres",
+  "auth": {
+    "enabled": true,
+    "mode": "mixed"
+  },
+  "rateLimit": {
+    "enabled": true,
+    "max": 60,
+    "window_ms": 60000
+  }
+}
+```
+
+## Migration Guide
+
+### From API Key Only to JWT
+
+1. **Add JWT configuration:**
+```bash
+AUTH_MODE=mixed          # Accept both during migration
+JWT_SECRET=<your-secret>
+```
+
+2. **Deploy and test:** Both API keys and JWT work
+
+3. **Switch clients to JWT** (gradual migration)
+
+4. **Once complete, switch to JWT-only:**
+```bash
+AUTH_MODE=jwt
+```
+
+### Disabling Features (Development Only)
+
+```bash
+# Disable authentication
+AUTH_ENABLED=false
+
+# Disable rate limiting
+RATE_LIMIT_ENABLED=false
+```
+
+**⚠️ Never disable in production!**
+
+## Error Response Reference
+
+| Status | Error Code | Description |
+|--------|------------|-------------|
+| 400 | `invalid_request` | Request body validation failed |
+| 400 | `invalid_query` | Query parameter validation failed |
+| 400 | `invalid_params` | Route parameter validation failed |
+| 401 | `unauthorized` | Missing or invalid credentials |
+| 403 | `forbidden` | Valid credentials but insufficient permissions |
+| 429 | `rate_limited` | Too many requests |
+
+## Security Considerations
+
+1. **JWT Secrets:**
+   - Use strong, random secrets (32+ characters)
+   - Generate with: `openssl rand -base64 32`
+   - Never commit secrets to git
+
+2. **Rate Limiting:**
+   - Adjust limits based on expected traffic
+   - Monitor for abuse patterns
+   - Consider using Redis for distributed systems
+
+3. **Validation:**
+   - All inputs are validated before processing
+   - Unknown fields are stripped
+   - Type coercion is explicit
+
+4. **Production:**
+   - Always enable `AUTH_ENABLED=true`
+   - Use `AUTH_MODE=jwt` or `mixed` for better security
+   - Set appropriate rate limits
+   - Use HTTPS in production
+
+## Troubleshooting
+
+### JWT Validation Fails
+
+**Problem:** 401 error with valid-looking JWT
+
+**Solutions:**
+1. Verify `JWT_SECRET` matches token generation secret
+2. Check token expiration (`exp` claim)
+3. Verify algorithm matches (`JWT_ALG`)
+4. Check token format: `Bearer <token>`
+
+### Rate Limit Errors
+
+**Problem:** Getting 429 even with few requests
+
+**Solutions:**
+1. Check if identity changes between requests (API key, JWT sub, IP)
+2. Verify `RATE_LIMIT_MAX` setting
+3. Check `RATE_LIMIT_WINDOW_MS` (default 1 minute)
+4. For development, set `RATE_LIMIT_ENABLED=false`
+
+### Validation Errors
+
+**Problem:** 400 errors on valid-looking requests
+
+**Solutions:**
+1. Check exact field requirements in schema
+2. Verify data types match (string vs number)
+3. Check for required vs optional fields
+4. Review detailed error in `details.fieldErrors`
+
+## Files Changed/Added
+
+### New Files
+- `apps/os-server/src/middleware/authJwt.ts`
+- `apps/os-server/src/middleware/auth.ts`
+- `apps/os-server/src/middleware/rateLimit.ts`
+- `apps/os-server/src/middleware/validate.ts`
+- `apps/os-server/src/schemas/runs.ts`
+- `apps/os-server/src/__tests__/auth.test.ts`
+- `apps/os-server/src/__tests__/validation.test.ts`
+- `apps/os-server/src/__tests__/rateLimit.test.ts`
+- `apps/os-server/ENV_VARIABLES.md`
+- `apps/os-server/JWT-RATE-LIMIT-VALIDATION.md` (this file)
+
+### Modified Files
+- `packages/shared/src/env.ts` - Added JWT and rate limit config
+- `apps/os-server/src/index.ts` - Updated all routes with new middleware
+- `infra/docker-compose.yml` - Added new environment variables
+
+## Dependencies
+
+All required dependencies were already installed:
+- `zod` - Schema validation
+- `fastify-rate-limit` - Rate limiting
+- Node.js `crypto` - JWT signature verification
+
+No new dependencies added! ✅
+

package/apps/os-server/PERSISTENCE-AND-AUTH.md

@@ -0,0 +1,227 @@
+# Persistent Runs + PostgreSQL + Authentication
+
+This document describes the implementation of persistent runs with PostgreSQL and API key authentication for the 4Runr Gateway.
+
+## Overview
+
+The Gateway now supports two persistence modes:
+- **Memory Mode** (`GATEWAY_PERSISTENCE=memory`): In-memory storage (no database)
+- **PostgreSQL Mode** (`GATEWAY_PERSISTENCE=postgres` or `persistent`): Persistent storage via Prisma/PostgreSQL
+
+All run endpoints are now protected by API key authentication (configurable via `AUTH_ENABLED`).
+
+## Architecture
+
+### Persistence Layer
+
+The persistence layer uses an adapter pattern with two implementations:
+
+1. **MemoryRunStore** (`apps/os-server/src/runs/memoryRunStore.ts`)
+   - Stores runs in a Map
+   - Data is lost on restart
+   - No database required
+
+2. **PostgresRunStore** (`apps/os-server/src/runs/postgresRunStore.ts`)
+   - Stores runs in PostgreSQL via Prisma
+   - Data persists across restarts
+   - Requires DATABASE_URL environment variable
+
+The store is selected automatically based on `GATEWAY_PERSISTENCE`:
+
+```typescript
+const runStore = getRunStore(); // Returns MemoryRunStore or PostgresRunStore
+```
+
+### Database Schema
+
+New Prisma models:
+
+- **Run**: Stores run data (id, name, status, input, output, logs, clientToken, tags, timestamps)
+- **ApiKey**: Stores API keys for authentication (key, label, revoked, timestamps)
+- **IdempotencyToken**: For idempotency token caching (future use)
+- **CacheEntry**: For general caching (future use)
+
+### Authentication
+
+API key authentication is implemented via middleware (`apps/os-server/src/middleware/authApiKey.ts`):
+
+- **Header**: `x-api-key: <token>`
+- **Validation**: Checks against `api_keys` table (PostgreSQL) or `API_KEY` env var (memory mode)
+- **Endpoints Protected**: All `/api/*` endpoints except `/health`, `/ready`, `/metrics`, and `/debug/*`
+
+## Environment Variables
+
+### Required (PostgreSQL Mode)
+
+```bash
+DATABASE_URL=postgresql://user:password@postgres:5432/dbname
+GATEWAY_PERSISTENCE=postgres  # or 'persistent'
+```
+
+### Optional
+
+```bash
+AUTH_ENABLED=true              # Enable/disable API key auth (default: true)
+API_KEY=your-key-here         # Single API key for memory mode or quick testing
+API_KEYS=key1,key2,key3       # Comma-separated list for memory mode
+```
+
+### Memory Mode
+
+```bash
+GATEWAY_PERSISTENCE=memory
+API_KEY=your-key-here         # Required if AUTH_ENABLED=true
+```
+
+## Setup
+
+### 1. Update Prisma Schema
+
+The schema has been updated to PostgreSQL. Run migrations:
+
+```bash
+cd apps/os-server
+pnpm db:generate
+pnpm db:migrate
+```
+
+### 2. Create API Keys
+
+For PostgreSQL mode, seed an API key:
+
+```bash
+pnpm seed:api-key
+# Or with custom key and label:
+pnpm seed:api-key my-custom-key "Production Key"
+```
+
+For memory mode, set the `API_KEY` environment variable:
+
+```bash
+export API_KEY=my-secret-key
+```
+
+### 3. Start Gateway
+
+```bash
+# PostgreSQL mode (default)
+export GATEWAY_PERSISTENCE=postgres
+export DATABASE_URL=postgresql://4runr:4runr_password@postgres:5432/4runr_gateway
+pnpm start
+
+# Memory mode
+export GATEWAY_PERSISTENCE=memory
+export API_KEY=my-secret-key
+pnpm start
+```
+
+## Usage
+
+### Create a Run (with auth)
+
+```bash
+curl -X POST http://localhost:3000/api/runs \
+  -H "Content-Type: application/json" \
+  -H "x-api-key: your-api-key" \
+  -d '{
+    "name": "My Run",
+    "input": {"foo": "bar"},
+    "tags": ["test"]
+  }'
+```
+
+### Get a Run
+
+```bash
+curl http://localhost:3000/api/runs/run-id \
+  -H "x-api-key: your-api-key"
+```
+
+### Start a Run
+
+```bash
+curl -X POST http://localhost:3000/api/runs/run-id/start \
+  -H "x-api-key: your-api-key"
+```
+
+## Testing
+
+### Disable Authentication (Development Only)
+
+```bash
+export AUTH_ENABLED=false
+```
+
+⚠️ **Warning**: Never disable authentication in production!
+
+### Test Memory Mode
+
+```bash
+export GATEWAY_PERSISTENCE=memory
+export API_KEY=test-key
+export AUTH_ENABLED=true
+pnpm start
+```
+
+### Test PostgreSQL Mode
+
+```bash
+export GATEWAY_PERSISTENCE=postgres
+export DATABASE_URL=postgresql://4runr:4runr_password@postgres:5432/4runr_gateway
+export AUTH_ENABLED=true
+# Seed an API key first
+pnpm seed:api-key test-key "Test Key"
+pnpm start
+```
+
+## Migration Guide
+
+### From In-Memory to PostgreSQL
+
+1. Set `GATEWAY_PERSISTENCE=postgres`
+2. Provide `DATABASE_URL`
+3. Run migrations: `pnpm db:migrate`
+4. Seed API keys: `pnpm seed:api-key`
+5. Restart gateway
+
+### From PostgreSQL to Memory
+
+1. Set `GATEWAY_PERSISTENCE=memory`
+2. Set `API_KEY` environment variable
+3. Restart gateway
+
+⚠️ **Note**: Data in PostgreSQL will remain but won't be accessible in memory mode.
+
+## File Structure
+
+```
+apps/os-server/
+├── src/
+│   ├── config/
+│   │   └── persistence.ts          # Persistence mode resolver
+│   ├── db/
+│   │   └── prisma.ts                # Prisma client singleton
+│   ├── runs/
+│   │   ├── types.ts                 # Run type definitions
+│   │   ├── runStore.ts              # RunStore interface
+│   │   ├── memoryRunStore.ts        # In-memory implementation
+│   │   ├── postgresRunStore.ts      # PostgreSQL implementation
+│   │   └── index.ts                 # Factory function
+│   ├── middleware/
+│   │   └── authApiKey.ts            # API key authentication
+│   └── index.ts                     # Main gateway server
+├── scripts/
+│   └── seed-api-key.ts              # API key seeding script
+└── prisma/
+    └── schema.prisma                # Database schema (PostgreSQL)
+```
+
+## Future Enhancements
+
+- [ ] Idempotency token store implementation
+- [ ] Cache entry store implementation
+- [ ] Bearer token authentication (JWT)
+- [ ] API key rotation
+- [ ] Rate limiting per API key
+- [ ] Audit logging for API key usage
+