agent-relay 1.1.0 → 1.2.0

package/docs/design/github-app-permissions.md

@@ -0,0 +1,264 @@
+# GitHub App Permissions
+
+This document describes the GitHub App permissions required for Agent Relay's features, particularly the CI failure webhook integration.
+
+## Overview
+
+Agent Relay uses a GitHub App to:
+1. Receive webhook events (installations, PRs, CI failures)
+2. Access repository code for cloning/syncing
+3. Create commits and push fixes
+4. Interact with PRs (comments, reviews)
+
+## Required Permissions
+
+### Repository Permissions
+
+| Permission | Access Level | Purpose |
+|------------|--------------|---------|
+| **Contents** | Read & Write | Clone repos, push commits, read files |
+| **Pull requests** | Read & Write | Read PR details, create/update PRs, comment |
+| **Checks** | Read | Receive check_run webhooks, read failure details |
+| **Actions** | Read | Receive workflow_run webhooks, read logs |
+| **Commit statuses** | Read | Read status checks, understand CI state |
+| **Metadata** | Read | Basic repo info (required for all apps) |
+
+### Organization Permissions
+
+| Permission | Access Level | Purpose |
+|------------|--------------|---------|
+| **Members** | Read | Identify organization members for access control |
+
+### Account Permissions
+
+| Permission | Access Level | Purpose |
+|------------|--------------|---------|
+| **Email addresses** | Read | User identification, notifications |
+
+## Webhook Events
+
+The following webhook events should be enabled:
+
+### Required Events
+
+| Event | Purpose |
+|-------|---------|
+| `installation` | Track app installations/uninstallations |
+| `installation_repositories` | Track repo access changes |
+| `check_run` | **CI failure detection** - triggers agent spawn |
+| `workflow_run` | Workflow-level failure tracking |
+| `push` | Detect new commits for sync |
+| `pull_request` | Track PR lifecycle |
+
+### Optional Events
+
+| Event | Purpose |
+|-------|---------|
+| `issues` | Future: issue-to-agent assignment |
+| `issue_comment` | Future: agent @mentions |
+| `pull_request_review` | Future: review request handling |
+| `check_suite` | Aggregate check status |
+
+## Configuration Steps
+
+### 1. Create GitHub App
+
+1. Go to GitHub Settings > Developer settings > GitHub Apps
+2. Click "New GitHub App"
+3. Fill in basic info:
+   - **Name**: Agent Relay (or your instance name)
+   - **Homepage URL**: Your dashboard URL
+   - **Webhook URL**: `https://your-domain.com/api/webhooks/github`
+
+### 2. Set Permissions
+
+Under "Permissions & events":
+
+**Repository permissions:**
+- Contents: Read and write
+- Pull requests: Read and write
+- Checks: Read-only
+- Actions: Read-only
+- Commit statuses: Read-only
+- Metadata: Read-only (default)
+
+**Organization permissions:**
+- Members: Read-only
+
+**Account permissions:**
+- Email addresses: Read-only
+
+### 3. Subscribe to Events
+
+Check the following events:
+- [x] Check run
+- [x] Workflow run
+- [x] Installation
+- [x] Installation and repositories
+- [x] Push
+- [x] Pull request
+
+### 4. Generate Keys
+
+1. Generate a private key (downloads .pem file)
+2. Note the App ID
+3. Generate a client secret for OAuth
+
+### 5. Configure Agent Relay
+
+Set environment variables:
+
+```bash
+# GitHub App credentials
+GITHUB_APP_ID=123456
+GITHUB_APP_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\n..."
+GITHUB_CLIENT_ID=Iv1.abc123
+GITHUB_CLIENT_SECRET=secret123
+
+# Webhook secret (generate a random string)
+GITHUB_WEBHOOK_SECRET=whsec_random_string_here
+```
+
+## Permission Rationale
+
+### Why Contents: Write?
+
+Agents need to push fixes to branches. This includes:
+- Creating new commits
+- Pushing to existing branches
+- Creating new branches for fixes
+
+### Why Checks: Read (not Write)?
+
+We only receive check failure events and read results. We don't:
+- Create our own checks
+- Update check status
+
+CI runs in GitHub Actions and creates its own checks.
+
+### Why Pull Requests: Write?
+
+Agents may need to:
+- Comment on PRs with fix summaries
+- Request reviews after fixes
+- Update PR descriptions
+
+### Why Actions: Read?
+
+For workflow_run events that provide:
+- Workflow-level failure context
+- Access to workflow logs (future)
+
+## Security Considerations
+
+### Webhook Secret
+
+Always configure a webhook secret:
+
+```typescript
+function verifyGitHubSignature(payload: string, signature: string, secret: string): boolean {
+  const expected = `sha256=${crypto
+    .createHmac('sha256', secret)
+    .update(payload)
+    .digest('hex')}`;
+  return crypto.timingSafeEqual(
+    Buffer.from(signature),
+    Buffer.from(expected)
+  );
+}
+```
+
+### Private Key Storage
+
+- Never commit the private key to version control
+- Use secure secret management (Vault, AWS Secrets Manager, etc.)
+- Rotate keys periodically
+
+### Installation Scope
+
+When users install the app:
+- Recommend "Only select repositories" over "All repositories"
+- Document which repos will be monitored
+- Allow easy un-installation
+
+### Token Expiry
+
+GitHub App installation tokens expire after 1 hour:
+- Cache tokens with expiry tracking
+- Refresh before expiration
+- Handle 401 errors with token refresh
+
+## Minimal Permissions Option
+
+For users who want minimal permissions:
+
+| Permission | Access | Notes |
+|------------|--------|-------|
+| Contents | Read | Can clone, cannot push |
+| Pull requests | Read | Can read PRs, cannot comment |
+| Checks | Read | Receive failures |
+
+With minimal permissions:
+- Agents can analyze failures but cannot push fixes
+- Manual intervention required for commits
+- Good for "notify only" mode
+
+## Events Flow
+
+```
+┌──────────────┐      webhook       ┌─────────────────┐
+│  GitHub CI   │ ──────────────────>│  Agent Relay    │
+│  (check_run  │                    │  /webhooks      │
+│   failed)    │                    └────────┬────────┘
+└──────────────┘                             │
+                                             │ verify signature
+                                             │ parse payload
+                                             ▼
+                                    ┌─────────────────┐
+                                    │  Record failure │
+                                    │  in database    │
+                                    └────────┬────────┘
+                                             │
+                                             │ spawn agent
+                                             ▼
+                                    ┌─────────────────┐
+                                    │  Agent fixes    │
+                                    │  and pushes     │
+                                    └────────┬────────┘
+                                             │
+                                             │ uses Contents:write
+                                             ▼
+                                    ┌─────────────────┐
+                                    │  GitHub CI      │
+                                    │  re-runs        │
+                                    └─────────────────┘
+```
+
+## Troubleshooting
+
+### Webhook Not Received
+
+1. Check webhook URL is correct and accessible
+2. Verify webhook secret matches configuration
+3. Check GitHub App webhook delivery logs
+4. Ensure firewall allows GitHub IPs
+
+### Permission Denied Errors
+
+1. Verify app is installed on the repository
+2. Check installation hasn't been suspended
+3. Confirm required permissions are granted
+4. Regenerate installation token
+
+### CI Events Not Triggering
+
+1. Verify `check_run` event is subscribed
+2. Check check is from a GitHub Action (not external CI)
+3. Ensure webhook URL is receiving events (check delivery logs)
+
+## References
+
+- [GitHub Apps documentation](https://docs.github.com/en/apps)
+- [Webhook events and payloads](https://docs.github.com/en/webhooks/webhook-events-and-payloads)
+- [GitHub App permissions](https://docs.github.com/en/rest/overview/permissions-required-for-github-apps)
+- [Check runs API](https://docs.github.com/en/rest/checks/runs)

package/docs/local-testing.md

@@ -0,0 +1,428 @@
+# Agent Relay Cloud - Local Testing Guide
+
+This guide explains how to run the complete Agent Relay Cloud stack locally for development and QA testing.
+
+## Overview
+
+The local testing environment simulates the full cloud deployment with:
+- **PostgreSQL** - Database for users, workspaces, metrics, crashes
+- **Redis** - Session storage and pub/sub messaging
+- **Cloud API Server** - Express.js control plane
+- **Daemon Simulators** - Simulated local daemons reporting metrics
+- **Integration Tests** - Comprehensive API tests
+
+## Prerequisites
+
+1. **Docker** (version 20.10+)
+2. **Docker Compose** (v2.0+)
+3. **Node.js** (v20+) - for running tests locally
+4. **Git** - for cloning the repository
+
+### Verify Prerequisites
+
+```bash
+docker --version          # Should be 20.10+
+docker compose version    # Should be 2.0+
+node --version           # Should be v20+
+```
+
+## Quick Start
+
+### Option 1: Full QA Suite (Recommended)
+
+Run the complete test suite with a single command:
+
+```bash
+./scripts/run-cloud-qa.sh
+```
+
+This will:
+1. Build all Docker images
+2. Start PostgreSQL and Redis
+3. Start the Cloud API server
+4. Start simulated daemons
+5. Run integration tests
+6. Clean up all containers
+
+### Option 2: Manual Setup
+
+For development and debugging, you may want to run components separately.
+
+#### Step 1: Start Infrastructure
+
+```bash
+# Start PostgreSQL and Redis
+docker compose -f docker-compose.test.yml up -d postgres redis
+
+# Verify they're healthy
+docker compose -f docker-compose.test.yml ps
+```
+
+#### Step 2: Start Cloud Server
+
+```bash
+# Start the cloud API server
+docker compose -f docker-compose.test.yml up -d cloud
+
+# Check logs
+docker compose -f docker-compose.test.yml logs -f cloud
+
+# Verify it's running
+curl http://localhost:3100/health
+```
+
+#### Step 3: Start Daemon Simulators
+
+```bash
+# Start simulated daemons that report metrics
+docker compose -f docker-compose.test.yml up -d daemon-simulator-1 daemon-simulator-2
+
+# View simulator logs
+docker compose -f docker-compose.test.yml logs -f daemon-simulator-1
+```
+
+#### Step 4: Run Tests
+
+```bash
+# Run integration tests in Docker
+docker compose -f docker-compose.test.yml --profile test run test-runner
+
+# Or run locally
+CLOUD_API_URL=http://localhost:3100 npm run test:integration
+```
+
+## Docker Compose Services
+
+### docker-compose.test.yml
+
+| Service | Port | Description |
+|---------|------|-------------|
+| `postgres` | 5433 | PostgreSQL database |
+| `redis` | 6380 | Redis for sessions/pub-sub |
+| `cloud` | 3100 | Cloud API server |
+| `daemon-simulator-1` | - | Simulated daemon (3 agents, memory growth) |
+| `daemon-simulator-2` | - | Simulated daemon (2 agents, normal) |
+| `daemon-simulator-crash` | - | Crash simulation daemon (profile: crash-test) |
+| `test-runner` | - | Integration test runner (profile: test) |
+
+### docker-compose.dev.yml
+
+For regular development (not testing):
+
+| Service | Port | Description |
+|---------|------|-------------|
+| `postgres` | 5432 | PostgreSQL database |
+| `redis` | 6379 | Redis |
+| `cloud` | 3000 | Cloud API + Dashboard |
+| `workspace` | 3888, 3889 | Example workspace (profile: workspace) |
+
+## Test Modes
+
+### Quick Smoke Test
+
+Fast validation that the stack is working:
+
+```bash
+./scripts/run-cloud-qa.sh --quick
+```
+
+### Full Integration Tests
+
+Complete test suite with all scenarios:
+
+```bash
+./scripts/run-cloud-qa.sh
+```
+
+### Keep Running After Tests
+
+Useful for debugging:
+
+```bash
+./scripts/run-cloud-qa.sh --keep
+```
+
+Then access:
+- Cloud API: http://localhost:3100
+- Health check: http://localhost:3100/health
+- Test status: http://localhost:3100/api/test/status
+
+### Show Logs
+
+View container logs after tests:
+
+```bash
+./scripts/run-cloud-qa.sh --logs
+```
+
+## Test Infrastructure
+
+### Daemon Simulator
+
+Located in `test/cloud/daemon-simulator.ts`, this simulates local daemons that:
+- Connect to the cloud API
+- Report agent memory metrics
+- Report crashes (configurable)
+- Report memory alerts
+
+Configuration via environment variables:
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `DAEMON_NAME` | test-daemon | Name of the daemon |
+| `CLOUD_API_URL` | http://localhost:3000 | Cloud API URL |
+| `AGENT_COUNT` | 3 | Number of agents to simulate |
+| `REPORT_INTERVAL_MS` | 10000 | Metrics report interval |
+| `SIMULATE_MEMORY_GROWTH` | false | Simulate memory leak |
+| `SIMULATE_CRASH` | false | Trigger crash after delay |
+| `CRASH_AFTER_SECONDS` | 60 | Delay before crash |
+
+### Test Helpers API
+
+In non-production mode, these endpoints are available:
+
+```bash
+# Check if test mode is enabled
+GET /api/test/status
+
+# Create a test user (bypasses OAuth)
+POST /api/test/create-user
+Body: { "email": "test@example.com", "name": "Test User" }
+
+# Create a test daemon with API key
+POST /api/test/create-daemon
+Body: { "name": "my-daemon", "machineId": "optional-machine-id" }
+
+# Cleanup test data
+DELETE /api/test/cleanup
+```
+
+### Integration Tests
+
+Located in `test/cloud/monitoring.integration.test.ts`:
+
+- Health check validation
+- Metrics reporting (authenticated/unauthenticated)
+- Crash reporting
+- Alert reporting
+- Dashboard API authentication
+- Multiple daemon scenarios
+- Alert escalation
+- Crash pattern detection
+
+## Running Tests Locally
+
+### Unit Tests (Fast)
+
+```bash
+# All unit tests
+npm test
+
+# Specific module
+npm test -- src/resiliency/
+
+# Watch mode
+npm test -- --watch
+```
+
+### Integration Tests
+
+```bash
+# Start the stack first
+docker compose -f docker-compose.test.yml up -d postgres redis cloud
+
+# Run integration tests
+CLOUD_API_URL=http://localhost:3100 npm run test:integration
+
+# Or with Docker
+docker compose -f docker-compose.test.yml --profile test run test-runner
+```
+
+### Coverage Report
+
+```bash
+npm run test:coverage
+```
+
+## Development Workflow
+
+### Making Changes
+
+1. Make code changes
+2. Run unit tests: `npm test`
+3. Start test stack: `docker compose -f docker-compose.test.yml up -d`
+4. Run integration tests: `npm run test:integration`
+5. Cleanup: `docker compose -f docker-compose.test.yml down -v`
+
+### Debugging Cloud Server
+
+```bash
+# Start with logs
+docker compose -f docker-compose.test.yml up cloud
+
+# Or attach to running container
+docker compose -f docker-compose.test.yml logs -f cloud
+
+# Shell into container
+docker compose -f docker-compose.test.yml exec cloud sh
+```
+
+### Database Access
+
+```bash
+# Connect to PostgreSQL
+docker compose -f docker-compose.test.yml exec postgres psql -U agent_relay -d agent_relay_test
+
+# View tables
+\dt
+
+# Query metrics
+SELECT * FROM agent_metrics ORDER BY recorded_at DESC LIMIT 10;
+
+# Query crashes
+SELECT * FROM agent_crashes ORDER BY crashed_at DESC LIMIT 10;
+```
+
+### Redis Access
+
+```bash
+# Connect to Redis
+docker compose -f docker-compose.test.yml exec redis redis-cli
+
+# View keys
+KEYS *
+
+# Monitor pub/sub
+SUBSCRIBE coordinator:messages
+```
+
+## Troubleshooting
+
+### Container Won't Start
+
+```bash
+# Check logs
+docker compose -f docker-compose.test.yml logs <service-name>
+
+# Rebuild images
+docker compose -f docker-compose.test.yml build --no-cache
+
+# Remove volumes and restart
+docker compose -f docker-compose.test.yml down -v
+docker compose -f docker-compose.test.yml up -d
+```
+
+### Database Connection Issues
+
+```bash
+# Verify PostgreSQL is healthy
+docker compose -f docker-compose.test.yml ps postgres
+
+# Check connection from cloud container
+docker compose -f docker-compose.test.yml exec cloud sh
+> nc -zv postgres 5432
+```
+
+### Port Conflicts
+
+If ports are already in use:
+
+```bash
+# Find what's using the port
+lsof -i :3100
+
+# Or change ports in docker-compose.test.yml
+```
+
+### Memory Issues
+
+Docker may run out of memory with many containers:
+
+```bash
+# Check Docker resource usage
+docker stats
+
+# Prune unused resources
+docker system prune -a
+
+# Increase Docker memory limit in Docker Desktop settings
+```
+
+## CI/CD Integration
+
+### GitHub Actions
+
+The test suite runs in GitHub Actions. See `.github/workflows/test.yml`:
+
+```yaml
+- name: Run Integration Tests
+  run: |
+    docker compose -f docker-compose.test.yml up -d postgres redis cloud
+    sleep 30
+    CLOUD_API_URL=http://localhost:3100 npm run test:integration
+```
+
+### Local CI Simulation
+
+```bash
+# Simulate CI environment
+./scripts/run-cloud-qa.sh
+```
+
+## Adding New Tests
+
+### Unit Tests
+
+1. Create `*.test.ts` file alongside the source
+2. Use Vitest patterns (describe, it, expect)
+3. Mock external dependencies
+
+### Integration Tests
+
+1. Add tests to `test/cloud/monitoring.integration.test.ts`
+2. Use the test helper API for setup
+3. Clean up test data in afterAll
+
+### New Simulator Scenarios
+
+1. Add new service to `docker-compose.test.yml`
+2. Configure via environment variables
+3. Use appropriate profile if optional
+
+## Reference
+
+### Environment Variables
+
+**Cloud Server:**
+- `NODE_ENV` - development/test/production
+- `DATABASE_URL` - PostgreSQL connection string
+- `REDIS_URL` - Redis connection string
+- `SESSION_SECRET` - Session encryption key
+- `RELAY_CLOUD_ENABLED` - Enable cloud features
+- `RELAY_MEMORY_MONITORING` - Enable memory monitoring
+
+**Test:**
+- `CLOUD_API_URL` - Cloud server URL for tests
+- `TEST_TIMEOUT` - Test timeout in milliseconds
+
+### Useful Commands
+
+```bash
+# Full QA suite
+./scripts/run-cloud-qa.sh
+
+# Quick test
+./scripts/run-cloud-qa.sh --quick
+
+# Keep running
+./scripts/run-cloud-qa.sh --keep
+
+# Cleanup only
+./scripts/run-cloud-qa.sh --cleanup
+
+# View all containers
+docker compose -f docker-compose.test.yml ps
+
+# Stop everything
+docker compose -f docker-compose.test.yml down -v
+```