octocode-cli 1.2.6 → 1.2.8

package/skills/octocode-research/docs/OVERVIEW.md

@@ -0,0 +1,564 @@
+# Octocode Research Skill - Research Document
+
+**Generated:** 2026-01-18  
+**Version:** 2.0.0
+
+---
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Architecture](#architecture)
+- [Core Components](#core-components)
+- [API Reference](#api-reference)
+- [Tool Registry](#tool-registry)
+- [Resilience Patterns](#resilience-patterns)
+- [Request Flow](#request-flow)
+- [Response Formats](#response-formats)
+- [File Structure](#file-structure)
+
+---
+
+## Overview
+
+The **octocode-research** skill is an Express.js HTTP server that wraps `octocode-mcp` tools as REST endpoints for code research. It listens to **localhost** on port **1987** and provides:
+
+| Feature | Description |
+|---------|-------------|
+| **Unified Tool API** | All tools via `POST /tools/call/:toolName` |
+| **MCP-compatible** | Prompts, tools, and system context following MCP patterns |
+| **Resilience** | Circuit breaker + retry with exponential backoff |
+| **Throttling** | Gradual slowdown for high-frequency requests |
+| **Session Telemetry** | Tool call tracking for analytics |
+
+### Key Design Principle
+
+> **All tools are executed via a unified `POST /tools/call/:toolName` endpoint, NOT individual GET routes.**
+
+---
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                          OCTOCODE RESEARCH SKILL                            │
+├─────────────────────────────────────────────────────────────────────────────┤
+│  HTTP Clients (curl/fetch)  ←──────→  Express Server (port 1987)            │
+│                                            │                                 │
+│                     ┌──────────────────────┼───────────────────────┐        │
+│                     │                      │                       │        │
+│                     ▼                      ▼                       ▼        │
+│              Local Tools            LSP Tools              GitHub Tools     │
+│              (ripgrep, fs)     (semantic analysis)           (API)          │
+│                                                                              │
+│              localSearchCode       lspGotoDefinition      githubSearchCode  │
+│              localGetFileContent   lspFindReferences      githubGetFile...  │
+│              localFindFiles        lspCallHierarchy       githubViewRepo... │
+│              localViewStructure                           githubSearchPR... │
+│                                                           packageSearch     │
+└─────────────────────────────────────────────────────────────────────────────┘
+                                     │
+                                     ▼
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                           EXTERNAL SYSTEMS                                   │
+│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐        │
+│  │ Filesystem  │  │ GitHub API  │  │ NPM/PyPI    │  │ LSP Server  │        │
+│  │  (ripgrep)  │  │  (Octokit)  │  │   APIs      │  │             │        │
+│  └─────────────┘  └─────────────┘  └─────────────┘  └─────────────┘        │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## Core Components
+
+### 1. Server (`src/server.ts`)
+
+The Express HTTP server that:
+- Initializes MCP content cache at startup via `initializeMcpContent()`
+- Initializes providers for GitHub token resolution via `initializeProviders()`
+- Mounts `/tools` and `/prompts` route handlers
+- Exposes `/health` endpoint for monitoring with circuit breaker states
+- Handles graceful shutdown with `SIGTERM`/`SIGINT` handlers
+
+**Key functions:**
+- `createServer()` - Creates and configures the Express app
+- `startServer()` - Starts the HTTP server on port 1987
+- `gracefulShutdown()` - Cleans up contexts and circuits on shutdown
+
+### 2. MCP Cache (`src/mcpCache.ts`)
+
+Singleton cache that loads tool metadata **ONCE** at startup:
+
+```typescript
+export async function initializeMcpContent(): Promise<CompleteMetadata> {
+  if (mcpContent) return mcpContent;
+  
+  initPromise = (async () => {
+    await initialize();
+    const content = await loadToolContent();
+    mcpContent = content;
+    return content;
+  })();
+  
+  return initPromise;
+}
+
+export function getMcpContent(): CompleteMetadata {
+  if (!mcpContent) {
+    throw new Error('mcpContent not initialized');
+  }
+  return mcpContent;
+}
+```
+
+### 3. Index (`src/index.ts`)
+
+Re-exports layer that maps `octocode-mcp` functions to skill-friendly names:
+
+| Export | Source |
+|--------|--------|
+| `githubSearchCode` | `searchMultipleGitHubCode` |
+| `githubGetFileContent` | `fetchMultipleGitHubFileContents` |
+| `localSearchCode` | `executeRipgrepSearch` |
+| `localGetFileContent` | `executeFetchContent` |
+| `lspGotoDefinition` | `executeGotoDefinition` |
+| `lspFindReferences` | `executeFindReferences` |
+| `lspCallHierarchy` | `executeCallHierarchy` |
+| `packageSearch` | `searchPackages` |
+
+### 4. Routes (`src/routes/`)
+
+| File | Mounted | Purpose |
+|------|---------|---------|
+| `tools.ts` | ✅ `/tools/*` | **Main API** - Tool discovery & execution |
+| `prompts.ts` | ✅ `/prompts/*` | Prompt discovery |
+| `local.ts` | ❌ | Handler logic (used in tests) |
+| `lsp.ts` | ❌ | Handler logic (used in tests) |
+| `github.ts` | ❌ | Handler logic (used in tests) |
+| `package.ts` | ❌ | Handler logic (used in tests) |
+
+---
+
+## API Reference
+
+### Discovery Endpoints
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/health` | GET | Server health, memory usage, circuit breaker states |
+| `/tools/list` | GET | List all tools (concise - name + description) |
+| `/tools/info/:toolName` | GET | Get tool JSON schema + hints |
+| `/tools/info` | GET | List all tools with details |
+| `/tools/system` | GET | Get system prompt (load FIRST) |
+| `/tools/initContext` | GET | Combined system prompt + all tool schemas (recommended for init) |
+| `/tools/metadata` | GET | Get raw metadata (advanced) |
+| `/prompts/list` | GET | List all prompts (MCP-compatible format) |
+| `/prompts/info/:promptName` | GET | Get prompt content and arguments |
+
+### Execution Endpoint
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/tools/call/:toolName` | **POST** | **Execute any tool** |
+
+**Request format:**
+```json
+{
+  "queries": [{
+    "mainResearchGoal": "Overall research objective",
+    "researchGoal": "This specific query's goal",
+    "reasoning": "Why this approach helps",
+    // ... tool-specific parameters
+  }]
+}
+```
+
+**Constraints:**
+- `queries` must be an array with 1-3 items
+- Each query must include research context parameters
+
+---
+
+## Tool Registry
+
+The `TOOL_REGISTRY` in `src/routes/tools.ts` maps tool names to their functions and resilience wrappers:
+
+```typescript
+const TOOL_REGISTRY: Record<string, ToolEntry> = {
+  // GitHub tools
+  githubSearchCode: { fn: githubSearchCode, resilience: withGitHubResilience, category: 'github' },
+  githubGetFileContent: { fn: githubGetFileContent, resilience: withGitHubResilience, category: 'github' },
+  githubViewRepoStructure: { fn: githubViewRepoStructure, resilience: withGitHubResilience, category: 'github' },
+  githubSearchRepositories: { fn: githubSearchRepositories, resilience: withGitHubResilience, category: 'github' },
+  githubSearchPullRequests: { fn: githubSearchPullRequests, resilience: withGitHubResilience, category: 'github' },
+
+  // Local tools
+  localSearchCode: { fn: localSearchCode, resilience: withLocalResilience, category: 'local' },
+  localGetFileContent: { fn: localGetFileContent, resilience: withLocalResilience, category: 'local' },
+  localFindFiles: { fn: localFindFiles, resilience: withLocalResilience, category: 'local' },
+  localViewStructure: { fn: localViewStructure, resilience: withLocalResilience, category: 'local' },
+
+  // LSP tools
+  lspGotoDefinition: { fn: lspGotoDefinition, resilience: withLspResilience, category: 'lsp' },
+  lspFindReferences: { fn: lspFindReferences, resilience: withLspResilience, category: 'lsp' },
+  lspCallHierarchy: { fn: lspCallHierarchy, resilience: withLspResilience, category: 'lsp' },
+
+  // Package tools
+  packageSearch: { fn: packageSearch, resilience: withPackageResilience, category: 'package' },
+};
+```
+
+### Available Tools (13 total)
+
+| Tool | Category | Description |
+|------|----------|-------------|
+| `localSearchCode` | Local | Search code with ripgrep |
+| `localGetFileContent` | Local | Read local file content |
+| `localFindFiles` | Local | Find files by pattern/metadata |
+| `localViewStructure` | Local | View directory tree |
+| `lspGotoDefinition` | LSP | Go to symbol definition |
+| `lspFindReferences` | LSP | Find all symbol references |
+| `lspCallHierarchy` | LSP | Get call hierarchy (incoming/outgoing) |
+| `githubSearchCode` | GitHub | Search code in repos |
+| `githubGetFileContent` | GitHub | Read file from repo |
+| `githubViewRepoStructure` | GitHub | View repo tree |
+| `githubSearchRepositories` | GitHub | Search repositories |
+| `githubSearchPullRequests` | GitHub | Search pull requests |
+| `packageSearch` | Package | Search npm/PyPI |
+
+---
+
+## Resilience Patterns
+
+### 1. Circuit Breaker (`src/utils/circuitBreaker.ts`)
+
+Prevents cascading failures with three states:
+
+```
+CLOSED (normal) ──[failures >= threshold]──► OPEN (reject all)
+       ▲                                          │
+       │                                     [timeout]
+       │                                          │
+       └──[successes >= threshold]─── HALF-OPEN ◄─┘
+                                        (probe)
+```
+
+**Default configuration:**
+```typescript
+const DEFAULT_CONFIG: CircuitBreakerConfig = {
+  failureThreshold: 3,    // Failures before opening
+  successThreshold: 2,    // Successes to close from half-open
+  resetTimeoutMs: 30000,  // Time before half-open probe
+};
+```
+
+**Per-service overrides:**
+
+| Service | Failure Threshold | Success Threshold | Timeout |
+|---------|-------------------|-------------------|---------|
+| LSP | 3 | 1 | 10s |
+| GitHub | 2 | 1 | 60s |
+
+**Key functions:**
+- `withCircuitBreaker(name, operation, fallback?)` - Execute with protection
+- `getCircuitState(name)` - Monitor circuit health
+- `configureCircuit(name, config)` - Customize thresholds
+- `getAllCircuitStates()` - Health dashboard (used in `/health`)
+
+### 2. Retry with Exponential Backoff (`src/utils/retry.ts`)
+
+```typescript
+const RETRY_CONFIGS = {
+  lsp: {
+    maxAttempts: 3,
+    initialDelayMs: 500,
+    maxDelayMs: 5000,
+    backoffMultiplier: 2,
+  },
+  github: {
+    maxAttempts: 3,
+    initialDelayMs: 1000,
+    maxDelayMs: 30000,
+    backoffMultiplier: 3,
+  },
+  package: {
+    maxAttempts: 3,
+    initialDelayMs: 1000,
+    maxDelayMs: 15000,
+    backoffMultiplier: 2,
+  },
+  local: {
+    maxAttempts: 2,
+    initialDelayMs: 100,
+    maxDelayMs: 1000,
+    backoffMultiplier: 2,
+  },
+};
+```
+
+### 3. Combined Resilience (`src/utils/resilience.ts`)
+
+Four pre-configured wrappers that combine circuit breaker + retry:
+
+```typescript
+export async function withGitHubResilience<T>(operation: () => Promise<T>, toolName: string): Promise<T>
+export async function withLspResilience<T>(operation: () => Promise<T>, toolName: string): Promise<T>
+export async function withLocalResilience<T>(operation: () => Promise<T>, toolName: string): Promise<T>
+export async function withPackageResilience<T>(operation: () => Promise<T>, toolName: string): Promise<T>
+```
+
+### 4. Readiness Check (`src/middleware/readiness.ts`)
+
+Middleware that verifies the server is ready to handle requests before processing them. Returns 503 Service Unavailable if the server is still initializing.
+
+---
+
+## Request Flow
+
+### Tool Execution Flow
+
+```
+POST /tools/call/localSearchCode
+Body: { "queries": [{ "pattern": "auth", "path": "src", ... }] }
+        │
+        ▼
+┌─────────────────────────────────────┐
+│  requestLogger Middleware           │
+│  → Log: tool, params, timestamp     │
+│  → Write to ~/.octocode/logs/       │
+└────────┬────────────────────────────┘
+         │
+         ▼
+┌─────────────────────────────────────┐
+│  Route Handler (routes/tools.ts)    │
+│  POST /tools/call/:toolName         │
+│                                     │
+│  1. Lookup tool in TOOL_REGISTRY    │
+│  2. Validate queries array (1-3)    │
+│  3. Get resilience wrapper          │
+└────────┬────────────────────────────┘
+         │
+         ▼
+┌─────────────────────────────────────┐
+│  Resilience Wrapper                 │
+│                                     │
+│  toolEntry.resilience(fn, toolName) │
+│   ├─ Circuit Breaker Check          │
+│   │   └─ OPEN? → Fail fast          │
+│   │   └─ CLOSED/HALF-OPEN? → Proceed│
+│   ├─ Retry with Backoff             │
+│   └─ Execute Tool Function          │
+└────────┬────────────────────────────┘
+         │
+         ▼
+┌─────────────────────────────────────┐
+│  octocode-mcp Tool Function         │
+│                                     │
+│  localSearchCode({ queries: [...] })│
+│   ├─ Execute ripgrep search         │
+│   ├─ Parse results                  │
+│   ├─ Apply hints                    │
+│   └─ Return CallToolResult          │
+└────────┬────────────────────────────┘
+         │
+         ▼
+┌─────────────────────────────────────┐
+│  Response Transformation            │
+│                                     │
+│  parseToolResponse(rawResult)       │
+│   ├─ Extract data, hints, research  │
+│   └─ Detect errors                  │
+└────────┬────────────────────────────┘
+         │
+         ▼
+HTTP Response (JSON)
+```
+
+---
+
+## Response Formats
+
+### Single Query Response
+
+```json
+{
+  "tool": "localSearchCode",
+  "success": true,
+  "data": {
+    "files": [...],
+    "totalMatches": 10,
+    "pagination": { "page": 1, "hasMore": false }
+  },
+  "hints": [
+    "Use lineHint for LSP tools",
+    "Consider narrowing search with path filter"
+  ],
+  "research": {
+    "mainResearchGoal": "Find authentication handlers",
+    "researchGoal": "Locate auth middleware",
+    "reasoning": "Understanding auth flow"
+  }
+}
+```
+
+### Bulk Query Response (2-3 queries)
+
+```json
+{
+  "tool": "localSearchCode",
+  "bulk": true,
+  "success": true,
+  "instructions": "Review results by status...",
+  "results": [
+    { "id": 1, "status": "hasResults", "data": {...}, "research": {...} },
+    { "id": 2, "status": "empty", "data": {}, "research": {...} }
+  ],
+  "hints": {
+    "hasResults": ["Follow up with LSP for semantic analysis"],
+    "empty": ["Try broader search pattern"],
+    "error": []
+  },
+  "counts": {
+    "total": 2,
+    "hasResults": 1,
+    "empty": 1,
+    "error": 0
+  }
+}
+```
+
+### Error Response
+
+```json
+{
+  "tool": "localSearchCode",
+  "success": false,
+  "data": {},
+  "hints": [
+    "Circuit breaker is open - retry in 30s",
+    "Consider using fallback tool"
+  ],
+  "research": {
+    "mainResearchGoal": "...",
+    "researchGoal": "...",
+    "reasoning": "..."
+  }
+}
+```
+
+---
+
+## File Structure
+
+```
+octocode-research/
+├── src/
+│   ├── server.ts              # Express server entry point
+│   ├── index.ts               # Re-exports from octocode-mcp
+│   ├── mcpCache.ts            # MCP client caching
+│   ├── routes/
+│   │   ├── tools.ts           # /tools/* - MAIN tool API
+│   │   ├── prompts.ts         # /prompts/* - prompt discovery
+│   │   ├── local.ts           # Handler logic (tests)
+│   │   ├── lsp.ts             # Handler logic (tests)
+│   │   ├── github.ts          # Handler logic (tests)
+│   │   └── package.ts         # Handler logic (tests)
+│   ├── middleware/
+│   │   ├── errorHandler.ts    # Error response formatting
+│   │   ├── logger.ts          # Request/response logging
+│   │   ├── queryParser.ts     # Zod validation
+│   │   └── readiness.ts       # Server readiness check
+│   ├── validation/
+│   │   ├── index.ts           # Schema exports
+│   │   ├── schemas.ts         # HTTP schemas (from octocode-mcp)
+│   │   └── httpPreprocess.ts  # Query string conversion
+│   ├── utils/
+│   │   ├── circuitBreaker.ts  # Circuit breaker pattern
+│   │   ├── colors.ts          # Console color functions
+│   │   ├── logger.ts          # File-based logging
+│   │   ├── resilience.ts      # Combined resilience wrappers
+│   │   ├── responseBuilder.ts # Research response formatting
+│   │   ├── responseFactory.ts # Safe data extraction
+│   │   ├── responseParser.ts  # MCP response parsing
+│   │   ├── retry.ts           # Retry with backoff
+│   │   └── routeFactory.ts    # Route handler factory
+│   ├── types/
+│   │   ├── express.d.ts       # Express type extensions
+│   │   ├── guards.ts          # Type guard functions
+│   │   ├── mcp.ts             # MCP protocol types
+│   │   ├── responses.ts       # Response type definitions
+│   │   └── toolTypes.ts       # Tool parameter types
+│   └── __tests__/
+│       ├── integration/       # Integration tests
+│       └── unit/              # Unit tests
+├── docs/
+│   ├── API_REFERENCE.md       # HTTP API reference
+│   ├── ARCHITECTURE.md        # Architecture documentation
+│   ├── FLOWS.md               # Flow diagrams
+│   └── OVERVIEW.md            # This document
+├── references/
+│   └── GUARDRAILS.md          # Safety guardrails
+├── scripts/
+│   ├── server.js              # Bundled server (tsdown)
+│   └── server.d.ts            # Type declarations
+├── SKILL.md                   # Skill definition for agents
+├── AGENTS.md                  # Development guide
+├── README.md                  # Quick start
+├── package.json
+├── tsconfig.json
+├── tsdown.config.ts
+└── vitest.config.ts
+```
+
+---
+
+## Quick Start
+
+```bash
+# Build
+npm run build
+
+# Start server
+npm start
+
+# Development with hot reload
+npm run dev
+
+# Health check
+curl http://localhost:1987/health
+
+# List tools
+curl http://localhost:1987/tools/list
+
+# Get tool schema
+curl http://localhost:1987/tools/info/localSearchCode
+
+# Execute tool
+curl -X POST http://localhost:1987/tools/call/localSearchCode \
+  -H "Content-Type: application/json" \
+  -d '{"queries": [{
+    "mainResearchGoal": "Find auth",
+    "researchGoal": "Search handlers",
+    "reasoning": "Testing",
+    "pattern": "authenticate",
+    "path": "/src"
+  }]}'
+```
+
+---
+
+## Related Documentation
+
+| Document | Purpose |
+|----------|---------|
+| [SKILL.md](https://github.com/bgauryy/octocode-mcp/blob/main/skills/octocode-research/SKILL.md) | Agent workflow guide |
+| [AGENTS.md](https://github.com/bgauryy/octocode-mcp/blob/main/skills/octocode-research/AGENTS.md) | Development guidelines |
+| [ARCHITECTURE.md](./ARCHITECTURE.md) | Full architecture details |
+| [FLOWS.md](./FLOWS.md) | Request flow diagrams |
+| [API_REFERENCE.md](./API_REFERENCE.md) | HTTP API reference |
+| [GUARDRAILS.md](https://github.com/bgauryy/octocode-mcp/blob/main/skills/octocode-research/references/GUARDRAILS.md) | Safety rules |
+
+---
+
+*Generated by Octocode Research Analysis*