miadi 1.0.14

package/MCP_REMOTE_SERVER_SPEC.md

@@ -0,0 +1,373 @@
+# Miadi MCP Remote Server Specification
+## Natural Language-Driven Software Development Prompt
+
+**Context**: Transform the existing Miadi MCP server from stdio-based local operation to HTTP-based remote server with OAuth 2.1 authentication, enabling claude.ai connector integration while preserving all 26 existing tools and tool selection capabilities.
+
+**Objective**: Create a production-ready remote MCP server that exposes Miadi Three-Pathway Agent System API through secure HTTP endpoints, supporting OAuth authentication flow with claude.ai callback integration and existing EH_TOKEN user validation.
+
+---
+
+## Core Requirements Specification
+
+### 1. Transport Layer Migration
+**From**: `StdioServerTransport` (JSON-RPC over stdin/stdout)  
+**To**: HTTP-based JSON-RPC server with Express.js framework
+
+**Implementation**: 
+- Express.js server listening on configurable port (default: 3000)
+- POST endpoint `/mcp/jsonrpc` handling MCP JSON-RPC protocol
+- Support for both single requests and batch requests
+- Proper CORS headers for cross-origin requests
+- Content-Type validation for `application/json`
+
+### 2. OAuth 2.1 Resource Server Implementation
+**Specification Compliance**: MCP OAuth 2.1 with Resource Server pattern (RFC8707)
+
+**Authentication Flow**:
+```
+1. Client discovers server via /.well-known/oauth-protected-resource
+2. Dynamic Client Registration (RFC7591) for automatic client setup
+3. Authorization Code + PKCE flow for secure token exchange
+4. Resource Indicators for token scoping to specific MCP server
+5. JWT token validation on every request
+6. EH_TOKEN integration for Miadi user database validation
+```
+
+**Endpoints Required**:
+- `/.well-known/oauth-protected-resource` - Resource Server Metadata
+- `/oauth/authorize` - Authorization endpoint (redirect to identity provider)
+- `/oauth/token` - Token exchange endpoint
+- `/oauth/userinfo` - User information endpoint for validation
+
+### 3. User Authentication Integration
+**Existing System**: Miadi API uses `EH_TOKEN` Bearer authentication  
+**Integration Pattern**: OAuth JWT contains user identifier → validate against EH_TOKEN system
+
+**User Validation Flow**:
+```
+1. OAuth token validated (signature, expiration, scope)
+2. Extract user identifier from JWT claims
+3. Call Miadi API with EH_TOKEN to validate user access
+4. Cache validation results (5-minute TTL)
+5. Reject requests for unauthorized users
+```
+
+### 4. Tool System Preservation
+**Requirement**: All existing functionality must be preserved
+- 26 tools across 6 categories (memory, session, capability, ai, workflow, forge)
+- Environment variable-based tool selection system
+- Complete Zod schemas and validation
+- Error handling and logging
+- Tool usage tracking and rate limiting
+
+### 5. Deployment Architecture
+**Local Development**: Ngrok tunnel for public endpoint exposure  
+**Production**: Cloud hosting with proper SSL/TLS termination
+
+**Ngrok Configuration**:
+```bash
+ngrok http 3000 --domain=<custom-domain>
+# Exposes local server at https://<custom-domain>.ngrok-free.app
+```
+
+**Environment Variables**:
+```bash
+# Existing Miadi API config
+EH_TOKEN=<miadi-api-token>
+EH_API_URL=<miadi-api-base-url>
+
+# OAuth configuration
+OAUTH_CLIENT_ID=<dynamic-or-configured>
+OAUTH_CLIENT_SECRET=<if-confidential-client>
+OAUTH_ISSUER=<identity-provider-url>
+JWT_SECRET=<token-validation-secret>
+
+# Server configuration
+MCP_PORT=3000
+MCP_HOST=0.0.0.0
+NGROK_AUTHTOKEN=<ngrok-token>
+
+# Tool selection (preserved)
+MIADI_TOOLS_ENABLED=<categories-or-tools>
+MIADI_TOOLS_DISABLED=<categories-or-tools>
+```
+
+---
+
+## Technical Implementation Specification
+
+### File Structure
+```
+src/
+├── index.ts                 # Main server entry (HTTP instead of stdio)
+├── http-transport.ts        # Express.js HTTP transport implementation
+├── oauth/
+│   ├── resource-server.ts   # OAuth 2.1 Resource Server implementation
+│   ├── middleware.ts        # Authentication middleware
+│   ├── user-validator.ts    # EH_TOKEN integration
+│   └── metadata.ts          # /.well-known/oauth-protected-resource
+├── tool-registry.ts         # Existing (preserved)
+├── utils.ts                 # Existing (preserved) 
+└── tools/                   # Existing tool modules (preserved)
+```
+
+### Core Components
+
+#### 1. HTTP Transport Layer (`http-transport.ts`)
+```typescript
+interface HttpMcpTransport {
+  // Express.js server with JSON-RPC handling
+  server: Express.Application
+  port: number
+  
+  // Methods
+  start(): Promise<void>
+  stop(): Promise<void>
+  handleJsonRpc(req: Request, res: Response): Promise<void>
+}
+```
+
+#### 2. OAuth Resource Server (`oauth/resource-server.ts`)
+```typescript
+interface OAuthResourceServer {
+  // OAuth 2.1 Resource Server implementation
+  validateToken(token: string): Promise<TokenValidationResult>
+  getUserFromToken(token: string): Promise<MiadiUser>
+  generateMetadata(): ResourceServerMetadata
+  
+  // Integration with Miadi user system
+  validateUserAccess(userId: string): Promise<boolean>
+}
+```
+
+#### 3. Authentication Middleware (`oauth/middleware.ts`)
+```typescript
+interface AuthMiddleware {
+  // Express middleware for token validation
+  authenticate(req: Request, res: Response, next: NextFunction): Promise<void>
+  
+  // User context injection
+  injectUser(req: AuthenticatedRequest): void
+}
+```
+
+### MCP JSON-RPC Protocol Adaptation
+
+**Request Format** (unchanged from stdio):
+```json
+{
+  "jsonrpc": "2.0",
+  "id": "request-id",
+  "method": "tools/call",
+  "params": {
+    "name": "miadi-get-memory",
+    "arguments": {
+      "key": "session:active",
+      "type": "auto"
+    }
+  }
+}
+```
+
+**HTTP Headers Required**:
+```
+Authorization: Bearer <jwt-token>
+Content-Type: application/json
+Accept: application/json
+```
+
+**Response Format** (unchanged):
+```json
+{
+  "jsonrpc": "2.0",
+  "id": "request-id",
+  "result": {
+    "content": [
+      {
+        "type": "text",
+        "text": "<tool-result-json>"
+      }
+    ],
+    "_meta": {
+      "tool": "miadi-get-memory",
+      "timestamp": "2025-01-08T..."
+    }
+  }
+}
+```
+
+---
+
+## Security Specifications
+
+### 1. OAuth 2.1 Security Requirements
+- **PKCE mandatory** for all authorization flows
+- **Resource Indicators** (RFC8707) for token scoping
+- **JWT validation** with signature verification and expiration checks
+- **HTTPS enforcement** in production (ngrok provides SSL termination)
+- **CORS policy** restricting origins to claude.ai domains
+
+### 2. Rate Limiting
+- **Per-user limits**: 100 requests per minute (existing implementation)
+- **Global limits**: 1000 requests per minute per server instance
+- **Tool-specific limits**: AI tools (10/min), others (50/min)
+
+### 3. User Authorization
+- **EH_TOKEN validation** for every authenticated request
+- **User scope validation** against Miadi user permissions
+- **Session invalidation** on user access revocation
+- **Audit logging** for all tool usage with user identification
+
+### 4. Error Security
+- **No sensitive data** in error responses
+- **Generic error messages** for authentication failures
+- **Request ID tracking** for debugging without exposing internals
+
+---
+
+## Claude.ai Integration Specification
+
+### Connector Registration
+**Claude.ai Settings** → **Connectors** → **Add Custom Connector**
+
+**Configuration Fields**:
+```
+Name: Miadi Agent System
+Base URL: https://<ngrok-domain>.ngrok-free.app
+Description: Access to Miadi Three-Pathway Agent System with memory operations, session management, and AI integration
+Icon: <miadi-logo-url>
+```
+
+### OAuth Discovery Flow
+1. Claude.ai fetches `/.well-known/oauth-protected-resource` from server
+2. Discovers authorization and token endpoints
+3. Initiates Dynamic Client Registration if supported
+4. Redirects user to authorization endpoint with PKCE challenge
+5. User authenticates with identity provider
+6. Authorization code returned to claude.ai callback
+7. Token exchange with PKCE verifier
+8. Claude.ai stores access token for MCP tool invocations
+
+### Tool Invocation Pattern
+```
+Claude.ai → HTTP POST /mcp/jsonrpc
+Headers: Authorization: Bearer <jwt>
+Body: MCP JSON-RPC tool call
+← Tool result with Miadi agent data
+```
+
+---
+
+## Testing and Validation Specification
+
+### 1. Unit Tests
+- OAuth token validation functions
+- User authorization against EH_TOKEN
+- HTTP transport JSON-RPC handling
+- Tool registry with HTTP context
+- Error handling and security edge cases
+
+### 2. Integration Tests
+- Complete OAuth flow with test identity provider
+- Tool execution through HTTP transport
+- User restriction enforcement
+- Rate limiting behavior
+- CORS and security headers
+
+### 3. End-to-End Tests
+- Ngrok tunnel establishment
+- Claude.ai connector registration
+- Full authentication and tool invocation flow
+- Error scenarios and recovery
+- Performance under load
+
+### 4. Security Testing
+- JWT token tampering attempts
+- Unauthorized user access attempts
+- Rate limiting bypass attempts
+- CORS policy violation tests
+- Input validation and injection tests
+
+---
+
+## Performance and Monitoring Specification
+
+### 1. Metrics Collection
+- **Request latency**: P50, P95, P99 response times
+- **Authentication latency**: OAuth validation timing
+- **Tool execution time**: Per-tool performance tracking
+- **Error rates**: Authentication failures, tool errors, rate limits
+- **User activity**: Tool usage patterns per user
+
+### 2. Logging Requirements
+- **Structured logging** with request IDs and user context
+- **Security events**: Authentication failures, unauthorized access
+- **Performance events**: Slow requests, rate limit hits
+- **Business events**: Tool usage, session management actions
+
+### 3. Health Endpoints
+- `/health` - Basic server health check
+- `/metrics` - Prometheus-compatible metrics
+- `/oauth/health` - OAuth provider connectivity
+- `/miadi/health` - Miadi API connectivity
+
+---
+
+## Migration and Rollback Specification
+
+### 1. Backward Compatibility
+- **Stdio transport preserved** for local development
+- **Environment variable** to select transport mode
+- **Tool registry unchanged** - no breaking changes to tool interface
+- **Configuration migration** from stdio to HTTP with sensible defaults
+
+### 2. Deployment Strategy
+- **Blue-green deployment** for production environments
+- **Canary rollout** for gradual user migration
+- **Rollback procedures** with automatic health check triggers
+- **Configuration validation** before server startup
+
+### 3. Migration Steps
+1. **Development**: Local HTTP server with ngrok tunnel
+2. **Staging**: Cloud deployment with test OAuth provider
+3. **Production**: Production OAuth integration and claude.ai connector
+4. **Validation**: User acceptance testing and performance verification
+5. **Rollout**: Gradual user migration with monitoring
+
+---
+
+## Implementation Acceptance Criteria
+
+### ✅ Core Functionality
+- [ ] HTTP server replaces stdio transport completely
+- [ ] All 26 tools function identically through HTTP
+- [ ] Tool selection system works with HTTP transport
+- [ ] Error handling and logging preserved
+
+### ✅ OAuth Integration
+- [ ] OAuth 2.1 Resource Server fully compliant
+- [ ] Dynamic Client Registration working
+- [ ] JWT token validation secure and performant
+- [ ] EH_TOKEN user validation integrated
+
+### ✅ Security Requirements
+- [ ] All requests require valid authentication
+- [ ] User authorization enforced for all tools
+- [ ] Rate limiting and CORS protection active
+- [ ] Security headers properly configured
+
+### ✅ Claude.ai Compatibility
+- [ ] Connector registration successful
+- [ ] OAuth flow completes without errors
+- [ ] All tools accessible through claude.ai interface
+- [ ] Performance acceptable for interactive use
+
+### ✅ Production Readiness
+- [ ] Ngrok tunnel stable and performant
+- [ ] Health checks and monitoring operational
+- [ ] Error handling graceful and informative
+- [ ] Documentation complete and accurate
+
+---
+
+This specification provides the complete natural language-driven blueprint for transforming the Miadi MCP server into a production-ready remote server with OAuth authentication and claude.ai integration while preserving all existing functionality and security requirements.