octocode-cli 1.2.6 → 1.2.7

package/skills/octocode-research/docs/SERVER_FLOWS.md

@@ -0,0 +1,631 @@
+# Server Flows - Octocode Research Server
+
+> Comprehensive documentation of startup, restart, health, and shutdown flows.
+> **v2.1.0**: Now powered by PM2 process manager.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Architecture Diagram](#architecture-diagram)
+- [1. Startup Flow](#1-startup-flow)
+- [2. Request Flow](#2-request-flow)
+- [3. Health Check Flow](#3-health-check-flow)
+- [4. PM2 Restart Strategies](#4-pm2-restart-strategies)
+- [5. Graceful Shutdown Flow](#5-graceful-shutdown-flow)
+- [Component Reference](#component-reference)
+- [PM2 Commands Reference](#pm2-commands-reference)
+
+---
+
+## Overview
+
+| Attribute | Value |
+|-----------|-------|
+| **Port** | 1987 |
+| **Version** | 2.1.0 |
+| **Process Manager** | PM2 |
+| **Restart Strategy** | Cron (hourly) + Memory threshold |
+| **Max Memory** | 500MB |
+| **Kill Timeout** | 10 seconds |
+
+### Key Design Patterns
+
+- **Warm Start**: Server accepts requests immediately, MCP initializes in background
+- **PM2 Process Management**: Automatic restarts, monitoring, and log management
+- **Cron Restart**: Server restarts hourly for memory hygiene (via PM2)
+- **Memory Guard**: Auto-restart if memory exceeds 500MB (via PM2)
+- **Circuit Breaker**: Protects external calls with failure thresholds
+- **Fire-and-Forget**: Background telemetry doesn't block responses
+- **Readiness Gate**: `/tools` and `/prompts` routes blocked until MCP ready
+- **Ready Signal**: PM2 waits for `process.send('ready')` before considering app online
+
+---
+
+## Architecture Diagram
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                           SERVER LIFECYCLE (PM2)                             │
+├─────────────────────────────────────────────────────────────────────────────┤
+│                                                                             │
+│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐                  │
+│  │   STARTUP    │───▶│   RUNNING    │───▶│  SHUTDOWN    │                  │
+│  └──────────────┘    └──────────────┘    └──────────────┘                  │
+│         │                   │                   ▲                           │
+│         │                   │                   │                           │
+│         ▼                   ▼                   │                           │
+│  ┌──────────────┐    ┌──────────────┐    ┌─────┴────────┐                  │
+│  │ INITIALIZING │    │  PM2 CRON    │───▶│ CRON_RESTART │                  │
+│  │ wait_ready   │    │ (every hour) │    │  (hourly)    │                  │
+│  └──────────────┘    └──────────────┘    └──────────────┘                  │
+│         │                                       │                           │
+│         │                                       │                           │
+│         │            ┌──────────────┐           │                           │
+│         │            │ MEMORY CHECK │───────────┤                           │
+│         │            │ (> 500MB)    │           │                           │
+│         │            └──────────────┘           │                           │
+│         │                                       │                           │
+│         └────────── process.send('ready') ─────▶│                           │
+│                                                                             │
+└─────────────────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                           REQUEST FLOW                                       │
+├─────────────────────────────────────────────────────────────────────────────┤
+│                                                                             │
+│  Request ──▶ [requestLogger] ──▶ [Route Handler]                            │
+│                                        │                                    │
+│                       ┌────────────────┴───────┐                            │
+│                       ▼                        ▼                            │
+│                 /health              /tools, /prompts                       │
+│                 (always)             [checkReadiness]                       │
+│                    │                        │                               │
+│                    ▼                        ▼                               │
+│               JSON Response          503 if !initialized                    │
+│                                      200 if ready                           │
+│                                                                             │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## 1. Startup Flow
+
+### Sequence Diagram
+
+```
+┌──────────┐  ┌───────────┐  ┌────────────┐  ┌──────────────┐  ┌───────────┐
+│   PM2    │  │ createSvr │  │ startSvr   │  │ Background   │  │  Ready    │
+└────┬─────┘  └─────┬─────┘  └─────┬──────┘  └──────┬───────┘  └─────┬─────┘
+     │              │              │                │                │
+     │ pm2 start    │              │                │                │
+     │─────────────▶│              │                │                │
+     │              │              │                │                │
+     │              │ createServer()                │                │
+     │              │─────────────▶│                │                │
+     │              │              │                │                │
+     │              │ ┌────────────┴────────────┐   │                │
+     │              │ │ 1. initializeLogger()   │   │                │
+     │              │ │ 2. initializeSession()  │   │                │
+     │              │ │ 3. Setup middleware     │   │                │
+     │              │ │ 4. Mount routes         │   │                │
+     │              │ │ 5. Error handlers       │   │                │
+     │              │ └────────────┬────────────┘   │                │
+     │              │              │                │                │
+     │              │◀─────────────│                │                │
+     │              │   app        │                │                │
+     │              │              │                │                │
+     │              │              │ app.listen(1987)               │
+     │              │              │───────────────▶│                │
+     │              │              │                │                │
+     │              │              │  'listening'   │                │
+     │              │              │◀───────────────│                │
+     │              │              │                │                │
+     │              │              │ ┌──────────────┴──────────────┐ │
+     │              │              │ │ BACKGROUND (async):         │ │
+     │              │              │ │ 1. initializeMcpContent()   │ │
+     │              │              │ │ 2. initializeProviders()    │ │
+     │              │              │ │ 3. process.send('ready')    │ │
+     │              │              │ │ 4. logSessionInit()         │ │
+     │              │              │ └──────────────┬──────────────┘ │
+     │              │              │                │                │
+     │◀──────────── process.send('ready') ─────────│                │
+     │  (PM2 marks app 'online')   │                │   MCP Ready!   │
+     │              │              │                │───────────────▶│
+     │              │              │                │                │
+```
+
+### Phase 1: Synchronous Setup (`createServer`)
+
+```typescript
+// Order matters! These run synchronously before server accepts requests
+
+1. initializeLogger()           // Creates log directory, sets up file logging
+2. initializeSession()          // Generates session ID for telemetry
+3. Express middleware stack:
+   ├── express.json()           // Parse JSON bodies
+   └── requestLogger            // Log all requests with timing
+4. Route mounting:
+   ├── GET /health              // Always available (no readiness check)
+   ├── /tools/*                 // checkReadiness middleware applied
+   └── /prompts/*               // checkReadiness middleware applied
+5. Error handlers:
+   ├── 404 handler              // Unknown routes
+   └── errorHandler middleware  // Catch-all error formatting
+```
+
+### Phase 2: HTTP Server Start (`startServer`)
+
+```typescript
+app.listen(PORT)
+  .on('listening', () => {
+    // Server is now accepting connections
+    // Status: 'initializing' (health check returns this)
+  });
+```
+
+### Phase 3: Background Initialization (Warm Start)
+
+```typescript
+// These run async AFTER server is listening
+// Requests to /tools/* return 503 until complete
+
+1. initializeMcpContent()       // Load MCP tools, schemas, prompts from octocode-mcp
+   ├── await initialize()       // Core MCP initialization
+   └── await loadToolContent()  // Cache tool metadata
+   
+2. initializeProviders()        // Initialize GitHub token, providers
+   └── Resolves token from env/keychain/CLI
+
+3. process.send('ready')        // Signal PM2 that app is ready
+   └── PM2 marks process as 'online'
+
+4. logSessionInit()             // Fire-and-forget telemetry
+   └── 5s timeout, errors queued
+```
+
+### State Transitions
+
+| State | Health Status | `/tools` Available | PM2 Status | Duration |
+|-------|---------------|-------------------|------------|----------|
+| Starting | N/A | No | starting | ~100ms |
+| Listening | `'initializing'` | 503 | waiting ready | ~1-3s |
+| Ready | `'ok'` | Yes | online | Until shutdown |
+
+---
+
+## 2. Request Flow
+
+### Middleware Chain
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                           REQUEST MIDDLEWARE CHAIN                           │
+├─────────────────────────────────────────────────────────────────────────────┤
+│                                                                             │
+│  Incoming Request                                                           │
+│         │                                                                   │
+│         ▼                                                                   │
+│  ┌──────────────────┐                                                       │
+│  │  express.json()  │  Parse JSON body                                      │
+│  └────────┬─────────┘                                                       │
+│           │                                                                 │
+│           ▼                                                                 │
+│  ┌──────────────────┐                                                       │
+│  │  requestLogger   │  Log request, attach x-request-id                     │
+│  └────────┬─────────┘                                                       │
+│           │                                                                 │
+│           ├───────────────────────────────────┐                             │
+│           │                                   │                             │
+│           ▼                                   ▼                             │
+│    ┌─────────────┐                   ┌────────────────┐                     │
+│    │  /health    │                   │ /tools, /prompts│                    │
+│    │  (direct)   │                   │ checkReadiness  │                    │
+│    └──────┬──────┘                   └───────┬────────┘                     │
+│           │                                  │                              │
+│           │                     ┌────────────┴────────────┐                 │
+│           │                     │                         │                 │
+│           │                     ▼                         ▼                 │
+│           │            MCP Initialized?            Not Initialized          │
+│           │                  YES                         NO                 │
+│           │                   │                          │                  │
+│           │                   ▼                          ▼                  │
+│           │            Route Handler            503 + retry hint            │
+│           │                   │                                             │
+│           ▼                   ▼                                             │
+│    ┌─────────────────────────────────────────────────────┐                  │
+│    │                   Response                          │                  │
+│    │  - requestLogger logs on 'finish' event             │                  │
+│    │  - logToolCall() fired async (non-blocking)         │                  │
+│    └─────────────────────────────────────────────────────┘                  │
+│                                                                             │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+### Request Logger Details
+
+```typescript
+// Attaches to response 'finish' event
+res.on('finish', () => {
+  const duration = Date.now() - start;
+  const status = res.statusCode;
+  
+  // Console log (colored)
+  console.log(status >= 400 ? errorLog(...) : resultLog(...));
+  
+  // File log (async, non-blocking)
+  if (req.path !== '/health') {
+    logToolCall({ tool, route, duration, success, requestId });
+  }
+});
+```
+
+### Readiness Check Middleware
+
+```typescript
+// Applied to /tools/* and /prompts/* routes
+export const checkReadiness = (_req, res, next) => {
+  if (!isMcpInitialized()) {
+    res.status(503).json({
+      success: false,
+      error: {
+        message: 'Server is initializing',
+        code: 'SERVER_INITIALIZING',
+        hint: 'Please retry in a few seconds',
+      },
+    });
+    return;
+  }
+  next();
+};
+```
+
+---
+
+## 3. Health Check Flow
+
+### Endpoint: `GET /health`
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                           HEALTH CHECK FLOW                                  │
+├─────────────────────────────────────────────────────────────────────────────┤
+│                                                                             │
+│  GET /health                                                                │
+│       │                                                                     │
+│       ├──▶ process.memoryUsage()         // Heap, RSS stats                 │
+│       │                                                                     │
+│       ├──▶ errorQueue.getRecent(5)       // Last 5 queued errors            │
+│       │                                                                     │
+│       ├──▶ isMcpInitialized()            // true = 'ok', false = 'init'     │
+│       │                                                                     │
+│       ├──▶ getAllCircuitStates()         // Circuit breaker health          │
+│       │                                                                     │
+│       └──▶ Response                                                         │
+│                                                                             │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+### Response Schema
+
+```typescript
+{
+  status: 'ok' | 'initializing',
+  port: 1987,
+  version: '2.1.0',
+  uptime: number,                    // seconds since process start
+  processManager: 'pm2',
+  memory: {
+    heapUsed: number,                // MB
+    heapTotal: number,               // MB
+    rss: number,                     // MB (Resident Set Size)
+  },
+  circuits: {
+    [circuitName]: {
+      state: 'closed' | 'open' | 'half-open',
+      failures: number,
+      lastFailure: Date | null,
+    }
+  },
+  errors: {
+    queueSize: number,
+    recentErrors: [{
+      timestamp: string,             // ISO 8601
+      context: string,
+      message: string,
+    }]
+  }
+}
+```
+
+### Health Check Behavior
+
+| Condition | Status | Notes |
+|-----------|--------|-------|
+| MCP not initialized | `'initializing'` | Background init in progress |
+| MCP initialized | `'ok'` | All routes available |
+| Circuit open | `'ok'` | Degraded but functional |
+| Errors in queue | `'ok'` | Errors are informational |
+
+---
+
+## 4. PM2 Restart Strategies
+
+### Overview
+
+PM2 manages all restart logic via `ecosystem.config.cjs`:
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                           PM2 RESTART STRATEGIES                             │
+├─────────────────────────────────────────────────────────────────────────────┤
+│                                                                             │
+│  ┌──────────────┐        ┌──────────────┐        ┌──────────────┐          │
+│  │ CRON RESTART │        │MEMORY RESTART│        │CRASH RESTART │          │
+│  │  (hourly)    │        │  (> 500MB)   │        │ (on error)   │          │
+│  └──────┬───────┘        └──────┬───────┘        └──────┬───────┘          │
+│         │                       │                       │                   │
+│         ▼                       ▼                       ▼                   │
+│  ┌─────────────────────────────────────────────────────────────┐           │
+│  │                     SIGINT → gracefulShutdown()             │           │
+│  │                                                             │           │
+│  │  1. stopCircuitCleanup()    ✅ Intervals cleared            │           │
+│  │  2. clearAllCircuits()      ✅ Circuit breakers reset       │           │
+│  │  3. server.close()          ✅ HTTP connections drained     │           │
+│  │  4. process.exit(0)         ✅ Clean exit                   │           │
+│  │                                                             │           │
+│  └─────────────────────────────────────────────────────────────┘           │
+│         │                                                                   │
+│         ▼                                                                   │
+│  ┌─────────────────────────────────────────────────────────────┐           │
+│  │                     PM2 Auto-Restart                        │           │
+│  │                                                             │           │
+│  │  - restart_delay: 1000ms                                    │           │
+│  │  - exp_backoff_restart_delay: 100ms (on crashes)            │           │
+│  │  - max_restarts: 10 (before stopping)                       │           │
+│  │                                                             │           │
+│  └─────────────────────────────────────────────────────────────┘           │
+│                                                                             │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+### Restart Triggers
+
+| Trigger | Condition | Behavior |
+|---------|-----------|----------|
+| **Cron** | Every hour (`0 * * * *`) | Graceful restart for memory hygiene |
+| **Memory** | RSS > 500MB | Automatic restart when threshold exceeded |
+| **Crash** | `process.exit(1)` | Restart with exponential backoff |
+| **Manual** | `pm2 restart` | Graceful reload |
+
+### Ecosystem Configuration
+
+```javascript
+// ecosystem.config.cjs
+module.exports = {
+  apps: [{
+    name: 'octocode-research',
+    script: './scripts/server.js',
+    
+    // Restart strategies
+    cron_restart: '0 * * * *',        // Every hour
+    max_memory_restart: '500M',        // Memory threshold
+    
+    // Graceful shutdown
+    kill_timeout: 10000,               // 10s before SIGKILL
+    wait_ready: true,                  // Wait for process.send('ready')
+    listen_timeout: 15000,             // Timeout for ready signal
+    
+    // Restart behavior
+    autorestart: true,
+    max_restarts: 10,
+    restart_delay: 1000,
+    exp_backoff_restart_delay: 100,
+  }]
+};
+```
+
+---
+
+## 5. Graceful Shutdown Flow
+
+### Triggers
+
+| Signal | Source |
+|--------|--------|
+| `SIGTERM` | `pm2 stop`, `pm2 restart`, `kill <pid>` |
+| `SIGINT` | Ctrl+C in terminal, PM2 graceful reload |
+
+### Sequence Diagram
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                           GRACEFUL SHUTDOWN FLOW                             │
+├─────────────────────────────────────────────────────────────────────────────┤
+│                                                                             │
+│  PM2 sends SIGINT (graceful) or SIGTERM                                     │
+│         │                                                                   │
+│         ▼                                                                   │
+│  ┌──────────────────┐                                                       │
+│  │ gracefulShutdown │                                                       │
+│  │    (signal)      │                                                       │
+│  └────────┬─────────┘                                                       │
+│           │                                                                 │
+│           ├──▶ stopCircuitCleanup()      // Clear circuit cleanup interval  │
+│           │         │                                                       │
+│           │         └──▶ clearInterval(cleanupIntervalId)                   │
+│           │                                                                 │
+│           ├──▶ clearAllCircuits()        // Reset all circuit breakers      │
+│           │         │                                                       │
+│           │         └──▶ circuits.clear(), configs.clear()                  │
+│           │                                                                 │
+│           └──▶ server.close()            // Stop accepting new connections  │
+│                     │                                                       │
+│           ┌─────────┴─────────┐                                            │
+│           │                   │                                            │
+│           ▼                   ▼                                            │
+│     Success (< 10s)    Timeout (10s)                                       │
+│           │                   │                                            │
+│           ▼                   ▼                                            │
+│     process.exit(0)    PM2 sends SIGKILL                                   │
+│           │                   │                                            │
+│           └─────────┬─────────┘                                            │
+│                     │                                                       │
+│                     ▼                                                       │
+│              PM2 Auto-Restart                                               │
+│                                                                             │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+### Cleanup Order
+
+```typescript
+function gracefulShutdown(signal: string): void {
+  console.log(`🛑 Received ${signal}. Starting graceful shutdown...`);
+
+  // 1. Stop periodic cleanup interval
+  stopCircuitCleanup();               // ✅ Circuit cleanup interval stopped
+
+  // 2. Clear circuit breaker state
+  clearAllCircuits();                 // ✅ Circuit breakers cleared
+
+  // 3. Close HTTP server (drain existing connections)
+  if (server) {
+    server.close((err) => {
+      if (err) {
+        console.error('❌ Error closing server:', err);
+        process.exit(1);
+      }
+      console.log('✅ HTTP server closed');
+      process.exit(0);               // PM2 handles restart
+    });
+  } else {
+    process.exit(0);
+  }
+}
+```
+
+### Exit Codes
+
+| Code | Meaning | PM2 Behavior |
+|------|---------|--------------|
+| 0 | Clean shutdown | Restart (unless `autorestart: false`) |
+| 1 | Error during shutdown | Restart with backoff |
+
+---
+
+## Component Reference
+
+### Files & Responsibilities
+
+| File | Purpose |
+|------|---------|
+| `ecosystem.config.cjs` | PM2 process configuration |
+| `server.ts` | Main entry, lifecycle management |
+| `mcpCache.ts` | MCP content singleton, initialization |
+| `index.ts` | Re-exports from octocode-mcp |
+| `middleware/readiness.ts` | Blocks routes until MCP ready |
+| `middleware/logger.ts` | Request/response logging |
+| `middleware/errorHandler.ts` | Error response formatting |
+| `utils/circuitBreaker.ts` | External call protection |
+| `utils/asyncTimeout.ts` | Fire-and-forget with timeout |
+| `utils/errorQueue.ts` | Error tracking for health |
+| `utils/logger.ts` | File-based logging |
+| `routes/tools.ts` | Tool execution endpoints |
+| `routes/prompts.ts` | Prompt discovery endpoints |
+
+### State Variables
+
+| Variable | Type | Purpose |
+|----------|------|---------|
+| `server` | `Server \| null` | HTTP server instance |
+| `mcpContent` | `CompleteMetadata \| null` | Cached MCP tool metadata |
+
+### Configuration (ecosystem.config.cjs)
+
+| Option | Value | Purpose |
+|--------|-------|---------|
+| `cron_restart` | `'0 * * * *'` | Hourly restart for memory hygiene |
+| `max_memory_restart` | `'500M'` | Memory threshold for restart |
+| `kill_timeout` | 10000 | Graceful shutdown timeout (ms) |
+| `wait_ready` | true | Wait for ready signal |
+| `listen_timeout` | 15000 | Ready signal timeout (ms) |
+| `max_restarts` | 10 | Max consecutive restarts |
+| `restart_delay` | 1000 | Delay between restarts (ms) |
+
+---
+
+## PM2 Commands Reference
+
+### NPM Scripts
+
+```bash
+# Start/Stop/Restart
+npm run pm2:start          # Start with PM2
+npm run pm2:stop           # Stop gracefully
+npm run pm2:restart        # Restart (full)
+npm run pm2:reload         # Reload (zero-downtime)
+npm run pm2:delete         # Remove from PM2
+
+# Monitoring
+npm run pm2:logs           # View logs (tail -f)
+npm run pm2:monit          # TUI dashboard
+```
+
+### Direct PM2 Commands
+
+```bash
+# Process management
+pm2 start ecosystem.config.cjs          # Start
+pm2 stop octocode-research              # Stop
+pm2 restart octocode-research           # Restart
+pm2 reload octocode-research            # Zero-downtime reload
+pm2 delete octocode-research            # Remove
+
+# Monitoring
+pm2 status                              # List all processes
+pm2 logs octocode-research              # View logs
+pm2 logs octocode-research --lines 100  # View last 100 lines
+pm2 monit                               # Interactive dashboard
+
+# Debugging
+pm2 describe octocode-research          # Detailed process info
+pm2 env octocode-research               # Environment variables
+pm2 reset octocode-research             # Reset restart counter
+
+# Log management
+pm2 flush                               # Clear all logs
+pm2 reloadLogs                          # Reload log files
+```
+
+### Development vs Production
+
+```bash
+# Development (direct Node, with watch)
+npm run dev                # tsx watch src/server.ts
+
+# Production (PM2 managed)
+npm run build              # Build TypeScript
+npm run pm2:start          # Start with PM2
+```
+
+---
+
+## Migration from v2.0.0
+
+### Removed Features (Handled by PM2)
+
+| Feature | Old Implementation | PM2 Equivalent |
+|---------|-------------------|----------------|
+| Idle restart | `checkIdleRestart()` | `cron_restart: '0 * * * *'` |
+| Idle timer | `lastRequestTime` | N/A |
+| Idle check interval | `setInterval(..., 300000)` | N/A |
+| Shutdown timeout | Manual `setTimeout` | `kill_timeout: 10000` |
+
+### New Features
+
+- **Ready Signal**: `process.send('ready')` for accurate startup tracking
+- **Memory Guard**: `max_memory_restart: '500M'`
+- **Exponential Backoff**: `exp_backoff_restart_delay: 100`
+- **PM2 Logs**: Disabled (app handles logging in `~/.octocode/logs/`)
+- **PM2 Monitoring**: `pm2 monit` dashboard