@agimon-ai/log-sink-mcp 0.2.0

package/README.md

@@ -0,0 +1,755 @@
+# log-sink-mcp
+
+AI-powered log analysis MCP server with HTTP ingestion and SQLite storage.
+
+## Overview
+
+`log-sink-mcp` provides a complete log analysis solution for AI assistants like Claude. It combines:
+
+- **HTTP Server**: REST endpoint for log ingestion from any application
+- **MCP Server**: 7 AI-optimized tools for querying and analyzing logs
+- **SQLite Database**: Fast local storage with FTS5 full-text search
+- **Real-time Analysis**: Query logs, trace distributed requests, analyze error patterns
+
+Perfect for debugging, monitoring, and understanding application behavior through AI-powered analysis.
+
+## Features
+
+- ✅ **HTTP Log Ingestion**: POST logs from any application via REST API
+- ✅ **7 MCP Tools**: Query, search, trace, analyze, and manage logs
+- ✅ **Full-Text Search**: Fast FTS5-powered search across messages and errors
+- ✅ **Distributed Tracing**: Timeline view for trace IDs and span relationships
+- ✅ **Error Analysis**: Pattern detection and error categorization
+- ✅ **Structured Logging**: JSON metadata, trace IDs, span IDs
+- ✅ **Auto-Migration**: Drizzle ORM with automatic schema migrations
+- ✅ **Type-Safe**: Full TypeScript coverage with Drizzle schema inference
+
+## Architecture
+
+```
+┌─────────────┐        HTTP POST         ┌─────────────────┐
+│ Application │ ───────────────────────> │  HTTP Server    │
+│  (Pino,     │     /logs endpoint       │  (Hono)         │
+│  Winston)   │                          │  Port 3100      │
+└─────────────┘                          └────────┬────────┘
+                                                  │
+                                                  ▼
+                                         ┌─────────────────┐
+                                         │  LogStorage     │
+                                         │  Service        │
+                                         └────────┬────────┘
+                                                  │
+                                                  ▼
+┌──────────────┐       MCP Protocol      ┌─────────────────┐
+│   Claude /   │ <─────────────────────> │   MCP Server    │
+│   AI Agent   │    7 Analysis Tools     │   (stdio)       │
+└──────────────┘                         └────────┬────────┘
+                                                  │
+                                                  ▼
+                                         ┌─────────────────┐
+                                         │  SQLite + FTS5  │
+                                         │  ./logs/        │
+                                         │  session.db     │
+                                         └─────────────────┘
+```
+
+## Singleton HTTP Server Management
+
+`log-sink-mcp` implements automatic singleton coordination to prevent port conflicts when multiple MCP servers run concurrently.
+
+### How It Works
+
+1. **Service Discovery**: When an MCP server starts, it checks `.agiflow/registry/log-sink-mcp-http_{env}.json` for an existing HTTP server
+2. **Health Check**: Pings `/health` endpoint to verify the registered server is alive
+3. **Reuse or Start**: If healthy, reuses existing server; otherwise starts a new one
+4. **PID Tracking**: Process IDs stored in `.pids/` for lifecycle management
+5. **Auto-Cleanup**: Registry and PID files removed on graceful shutdown
+
+### Service Registry Schema
+
+```json
+{
+  "name": "log-sink-mcp-http",
+  "port": 3100,
+  "host": "localhost",
+  "environment": "development",
+  "pid": 12345,
+  "status": "running",
+  "startedAt": "2026-01-15T10:00:00.000Z",
+  "metadata": {
+    "healthCheckUrl": "http://localhost:3100/health",
+    "dbPath": "./logs/session.db"
+  }
+}
+```
+
+### Management Commands
+
+```bash
+# Check status of HTTP server
+bun run src/cli.ts status
+
+# Stop HTTP server
+bun run src/cli.ts stop
+
+# Start with auto-coordination
+bun run src/cli.ts start
+```
+
+## Installation
+
+```bash
+# From monorepo root
+pnpm install
+
+# Or install globally
+pnpm add -g log-sink-mcp
+```
+
+## Quick Start
+
+### 1. Start the Servers
+
+```bash
+# Start both HTTP and MCP servers (recommended)
+bun run src/cli.ts start
+
+# Or start only HTTP server
+bun run src/cli.ts start --http-only
+
+# Or start only MCP server (stdio transport)
+bun run src/cli.ts start --mcp-only
+
+# With custom configuration
+bun run src/cli.ts start --port 3100 --db-path ./logs/session.db
+```
+
+The `start` command uses singleton pattern - multiple MCP servers will share a single HTTP server automatically.
+
+### 2. Configure MCP Server
+
+Add to your `mcp-config.yaml`:
+
+```yaml
+mcpServers:
+  log-sink:
+    type: stdio
+    command: bun
+    args:
+      - ./packages/log-sink-mcp/src/cli.ts
+      - mcp-serve
+      - --db-path
+      - ./logs/session.db
+    disabled: false
+```
+
+### 3. Send Logs
+
+```bash
+# Send a test log
+curl -X POST http://localhost:3100/logs \
+  -H "Content-Type: application/json" \
+  -d '{
+    "logs": [{
+      "level": "info",
+      "message": "Application started",
+      "service": "my-app",
+      "metadata": {"version": "1.0.0"}
+    }]
+  }'
+```
+
+### 4. Query with AI
+
+Ask Claude:
+
+> "Query all error logs from the database"
+>
+> "Show me the trace timeline for trace ID abc123"
+>
+> "Analyze error patterns in the logs"
+
+## HTTP API Reference
+
+### POST /logs
+
+Ingest log entries in batch.
+
+**Request Body:**
+
+```typescript
+{
+  logs: Array<{
+    // Required fields
+    level: 'debug' | 'info' | 'warn' | 'error';
+    message: string;
+    service: string;
+
+    // Optional fields
+    timestamp?: string;        // ISO 8601, defaults to now
+    traceId?: string;          // 32-char hex for distributed tracing
+    spanId?: string;           // 16-char hex for span identification
+    parentSpanId?: string;     // 16-char hex for parent span
+    hostname?: string;
+    pid?: number;
+    metadata?: Record<string, any>;
+    errorType?: string;
+    errorMessage?: string;
+    errorStack?: string;
+  }>
+}
+```
+
+**Response (201 Created):**
+
+```json
+{
+  "success": true,
+  "count": 5,
+  "message": "Successfully inserted 5 logs"
+}
+```
+
+**Response (400 Bad Request):**
+
+```json
+{
+  "success": false,
+  "error": "Validation failed",
+  "details": [...]
+}
+```
+
+### GET /health
+
+Health check endpoint.
+
+**Response (200 OK):**
+
+```json
+{
+  "status": "healthy",
+  "service": "log-sink-mcp",
+  "timestamp": "2026-01-15T10:00:00.000Z"
+}
+```
+
+## MCP Tools Reference
+
+### 1. `query_logs`
+
+Filter logs by level, service, or trace ID.
+
+**Parameters:**
+- `level?`: string - Filter by log level (debug, info, warn, error)
+- `service?`: string - Filter by service name
+- `traceId?`: string - Filter by trace ID
+- `limit?`: number - Maximum results (default: 100)
+
+**Example:**
+```
+Query all error logs from user-service
+```
+
+**Returns:** JSON array of matching log entries with timestamps, messages, metadata.
+
+---
+
+### 2. `search_logs`
+
+Full-text search across log messages and error messages.
+
+**Parameters:**
+- `searchQuery`: string (required) - Text to search for
+- `limit?`: number - Maximum results (default: 50)
+
+**Example:**
+```
+Search logs for "database connection timeout"
+```
+
+**Returns:** Matching logs with search relevance, count of results.
+
+---
+
+### 3. `get_trace_timeline`
+
+Get chronological timeline for a distributed trace.
+
+**Parameters:**
+- `traceId`: string (required) - Trace ID to retrieve
+
+**Example:**
+```
+Show me the trace timeline for trace ID a1b2c3d4e5f6789012345678901234ab
+```
+
+**Returns:** Ordered sequence of log entries for the trace, showing service call flow.
+
+---
+
+### 4. `analyze_errors`
+
+Analyze error patterns and categorize by error type.
+
+**Parameters:**
+- `limit?`: number - Maximum errors to analyze (default: 100)
+
+**Example:**
+```
+Analyze error patterns in the logs
+```
+
+**Returns:** Error categories, frequencies, example stack traces, common patterns.
+
+---
+
+### 5. `get_log_stats`
+
+Get statistics grouped by log level.
+
+**Parameters:** None
+
+**Example:**
+```
+Show me log statistics
+```
+
+**Returns:** Count per log level (debug, info, warn, error), total count.
+
+---
+
+### 6. `get_services`
+
+List all unique services that have logged.
+
+**Parameters:** None
+
+**Example:**
+```
+What services are logging to the database?
+```
+
+**Returns:** Array of service names with log counts.
+
+---
+
+### 7. `clear_logs`
+
+Delete all logs from the database.
+
+**Parameters:** None
+
+**Example:**
+```
+Clear all logs from the database
+```
+
+**Returns:** Confirmation with count of deleted entries.
+
+⚠️ **Warning:** This operation cannot be undone.
+
+## Integration Examples
+
+### Pino Logger
+
+```typescript
+import pino from 'pino';
+
+const logger = pino({
+  transport: {
+    target: 'pino-http-send',
+    options: {
+      url: 'http://localhost:3100/logs',
+      method: 'POST',
+    },
+  },
+});
+
+logger.info({ traceId: 'abc123', userId: '456' }, 'User logged in');
+```
+
+### Winston Logger
+
+```typescript
+import winston from 'winston';
+
+const logger = winston.createLogger({
+  transports: [
+    new winston.transports.Http({
+      host: 'localhost',
+      port: 3100,
+      path: '/logs',
+    }),
+  ],
+});
+
+logger.error('Database connection failed', {
+  service: 'api',
+  errorType: 'ConnectionError'
+});
+```
+
+### Custom Fetch
+
+```typescript
+async function sendLog(level: string, message: string, metadata?: any) {
+  await fetch('http://localhost:3100/logs', {
+    method: 'POST',
+    headers: { 'Content-Type': 'application/json' },
+    body: JSON.stringify({
+      logs: [{
+        level,
+        message,
+        service: 'my-service',
+        metadata,
+        timestamp: new Date().toISOString(),
+      }],
+    }),
+  });
+}
+
+await sendLog('info', 'Request processed', { duration: 245, status: 200 });
+```
+
+## Configuration
+
+### CLI Options
+
+#### `start` command (Recommended)
+
+Start HTTP and/or MCP servers with automatic singleton coordination.
+
+```bash
+bun run src/cli.ts start [options]
+```
+
+Options:
+- `--port <number>`: HTTP server port (default: 3100)
+- `--db-path <path>`: SQLite database path (default: ./logs/session.db)
+- `--in-memory`: Use in-memory database for testing (default: false)
+- `--http-only`: Start only the HTTP server
+- `--mcp-only`: Start only the MCP server (stdio transport)
+
+Examples:
+```bash
+# Start both servers with defaults
+bun run src/cli.ts start
+
+# Production mode with persistent database
+bun run src/cli.ts start --port 3100 --db-path ./logs/production.db
+
+# Testing with in-memory database
+bun run src/cli.ts start --in-memory
+```
+
+#### `status` command
+
+Show diagnostic information about running services.
+
+```bash
+bun run src/cli.ts status
+```
+
+Displays:
+- HTTP server status (running/stopped)
+- Process ID and port
+- Database file path and size
+- Health check URL
+
+#### `stop` command
+
+Gracefully stop HTTP server and cleanup registry/PID files.
+
+```bash
+bun run src/cli.ts stop
+```
+
+Behavior:
+- Sends SIGTERM to HTTP server process
+- Removes `.agiflow/registry/` entries
+- Cleans up `.pids/` files
+
+#### `http-serve` command
+
+Manually start HTTP log ingestion server (for advanced use).
+
+```bash
+bun run src/cli.ts http-serve [options]
+```
+
+Options:
+- `--port <number>`: HTTP server port (default: 3000)
+- `--db-path <path>`: SQLite database path (default: ./logs.db)
+- `--in-memory`: Use in-memory database (default: false)
+
+**Note:** Prefer using `start` command for automatic singleton coordination.
+
+#### `mcp-serve` command
+
+Manually start MCP server (for advanced use or custom transports).
+
+```bash
+bun run src/cli.ts mcp-serve [options]
+```
+
+Options:
+- `--type <type>`: Transport type: stdio, http, or sse (default: stdio)
+- `--port <number>`: Port for http/sse transport (default: 3000)
+- `--host <host>`: Host for http/sse transport (default: localhost)
+- `--db-path <path>`: SQLite database path (default: ./logs/session.db)
+- `--in-memory`: Use in-memory database (default: false)
+- `--cleanup`: Stop HTTP server on MCP shutdown (stdio only, default: false)
+
+Examples:
+```bash
+# stdio transport (for Claude Desktop)
+bun run src/cli.ts mcp-serve --type stdio
+
+# HTTP transport
+bun run src/cli.ts mcp-serve --type http --port 3001
+
+# SSE transport
+bun run src/cli.ts mcp-serve --type sse --port 3002
+```
+
+**Note:** For stdio transport, prefer using `start --mcp-only` for automatic HTTP coordination.
+
+#### `logs` command
+
+Run the same analysis capabilities exposed by the package MCP tools directly from the shell.
+
+```bash
+bun run src/cli.ts logs <subcommand> [options]
+```
+
+Subcommands:
+- `query`: Equivalent to `query_logs`
+- `search`: Equivalent to `search_logs`
+- `trace`: Equivalent to `get_trace_timeline`
+- `analyze-errors`: Equivalent to `analyze_errors`
+- `stats`: Equivalent to `get_log_stats`
+- `services`: Equivalent to `get_services`
+- `clear`: Equivalent to `clear_logs`
+
+All subcommands support:
+- `--db-path <path>`: SQLite database path (default: `./logs/session.db`)
+- `--in-memory`: Use an in-memory database
+
+Examples:
+
+```bash
+# Query error logs
+bun run src/cli.ts logs query --level error --db-path ./logs/session.db
+
+# Search for timeout failures
+bun run src/cli.ts logs search "timeout OR retry" --service api-gateway user-service
+
+# Inspect one trace
+bun run src/cli.ts logs trace a1b2c3d4e5f6789012345678901234ab
+
+# Analyze grouped errors in a time window
+bun run src/cli.ts logs analyze-errors \
+  --start-time 2026-01-15T10:00:00Z \
+  --end-time 2026-01-15T11:00:00Z
+
+# View service names discovered in the log database
+bun run src/cli.ts logs services
+```
+
+### Environment Variables
+
+None required. All configuration via CLI flags.
+
+## Development
+
+### Setup
+
+```bash
+# Install dependencies
+pnpm install
+
+# Run development server
+pnpm dev
+
+# Run HTTP server in dev mode
+bun run src/cli.ts http-serve --port 3100
+```
+
+### Project Structure
+
+```
+log-sink-mcp/
+├── src/
+│   ├── cli.ts                    # CLI entry point
+│   ├── commands/                 # CLI commands
+│   │   ├── http-serve.ts        # HTTP server command
+│   │   └── mcp-serve.ts         # MCP server command
+│   ├── container/               # InversifyJS DI container
+│   ├── models/                  # Drizzle schemas
+│   │   └── schema.ts            # Log table schema
+│   ├── server/
+│   │   ├── http.ts              # Hono HTTP server
+│   │   └── index.ts             # MCP server
+│   ├── services/                # Business logic
+│   │   ├── LogStorageService.ts
+│   │   ├── LogQueryService.ts
+│   │   ├── LogSearchService.ts
+│   │   └── LogRetentionService.ts
+│   ├── tools/                   # MCP tools
+│   │   ├── QueryLogsTool.ts
+│   │   ├── SearchLogsTool.ts
+│   │   ├── GetTraceTimelineTool.ts
+│   │   ├── AnalyzeErrorsTool.ts
+│   │   ├── GetLogStatsTool.ts
+│   │   ├── GetServicesTool.ts
+│   │   └── ClearLogsTool.ts
+│   └── types/                   # TypeScript types
+├── migrations/                  # Drizzle migrations
+├── tests/
+│   ├── integration/             # Integration tests
+│   ├── services/                # Service unit tests
+│   ├── tools/                   # Tool unit tests
+│   └── commands/                # Command tests
+└── README.md
+```
+
+### Testing
+
+```bash
+# Run all tests
+pnpm test
+
+# Run integration tests only
+pnpm test integration
+
+# Run with coverage
+pnpm test --coverage
+```
+
+**Test Coverage:**
+- HTTP server integration tests (8 tests)
+- MCP tools integration tests (17 tests)
+- Command tests (5 tests)
+
+### Adding New Tools
+
+1. Create tool class in `src/tools/`
+2. Implement `Tool` interface with `getDefinition()` and `execute()`
+3. Register in `src/server/index.ts`
+4. Add integration tests in `tests/integration/`
+
+See existing tools for patterns.
+
+## Troubleshooting
+
+### Check Service Status
+
+Use the `status` command to diagnose issues:
+
+```bash
+bun run src/cli.ts status
+```
+
+This shows:
+- Whether HTTP server is running
+- Process ID and port
+- Database path and size
+- Health check URL
+
+### Logs not appearing
+
+1. Check HTTP server is running:
+   ```bash
+   # Using status command
+   bun run src/cli.ts status
+
+   # Or direct health check
+   curl http://localhost:3100/health
+   ```
+
+2. Verify log format matches API schema
+3. Check database path is accessible
+4. Review HTTP server logs for errors
+
+### MCP tools not working
+
+1. Verify database file exists and has logs:
+   ```bash
+   bun run src/cli.ts status
+   ```
+
+2. Check `--db-path` matches between HTTP and MCP servers
+3. Restart MCP server after configuration changes:
+   ```bash
+   bun run src/cli.ts stop
+   bun run src/cli.ts start
+   ```
+
+### Database locked errors
+
+SQLite uses WAL mode for better concurrency, but:
+- Don't run multiple HTTP servers on same database
+- Ensure file permissions allow writes
+- Use separate databases for testing
+
+### Finding and Killing HTTP Server
+
+If you need to manually stop the HTTP server:
+
+```bash
+# Graceful stop (recommended)
+bun run src/cli.ts stop
+
+# Find process manually
+cat .pids/log-sink-mcp-http-development.pid
+
+# Kill process (if graceful stop fails)
+kill $(cat .pids/log-sink-mcp-http-development.pid)
+
+# Force kill (last resort)
+kill -9 $(cat .pids/log-sink-mcp-http-development.pid)
+```
+
+### Stale Registry Entries
+
+If the registry file exists but server isn't running:
+
+```bash
+# Clean up stale registry
+rm .agiflow/registry/log-sink-mcp-http_development.json
+rm .pids/log-sink-mcp-http-development.pid
+
+# Restart
+bun run src/cli.ts start
+```
+
+The health check will automatically detect stale registries and start a new server.
+
+### Port Already in Use
+
+If port 3100 is already taken:
+
+```bash
+# Use different port
+bun run src/cli.ts start --port 3101
+
+# Or find what's using the port
+lsof -i :3100
+
+# Kill the process using the port
+kill $(lsof -t -i :3100)
+```
+
+## License
+
+MIT
+
+## Contributing
+
+This package is part of the Agiflow monorepo. Contributions welcome!
+
+1. Follow coding standards in `/docs/CODING_STANDARDS.md`
+2. Use architect MCP before modifying code
+3. Add tests for new features
+4. Run `pnpm test` before committing