mcpgraph 0.1.5 → 0.1.7

package/docs/implementation.md

@@ -0,0 +1,377 @@
+# mcpGraph Implementation
+
+This document describes the implementation of the mcpGraph MCP server with graph execution capabilities. The visual editor/UX is deferred to a later phase.
+
+## Overview
+
+The implementation creates a working MCP server that can:
+1. Parse YAML configuration files
+2. Expose MCP tools defined in the configuration
+3. Execute directed graphs of nodes when tools are called
+4. Handle data transformation and routing between nodes
+
+## Phase 1: Foundation & Configuration Parsing
+
+### 1.1 Project Setup & Dependencies
+
+**Implemented:**
+- TypeScript project initialized
+- Logger (stderr only) implemented in `src/logger.ts`
+- All dependencies added:
+  - `@modelcontextprotocol/sdk` - MCP SDK
+  - `jsonata` - Data transformation
+  - `json-logic-js` - Conditional routing
+  - `js-yaml` - YAML parsing
+  - `zod` - Schema validation
+
+**Note:** A custom execution loop was implemented to provide full control over execution flow and enable future introspection/debugging capabilities.
+
+### 1.2 Type Definitions
+
+**File: `src/types/config.ts`**
+
+All TypeScript interfaces defined:
+- `McpGraphConfig` - Root configuration structure
+- `ServerMetadata` - MCP server metadata
+- `ToolDefinition` - Tool definition with input/output schemas (entry/exit nodes are defined in nodes with `tool` field)
+- `NodeDefinition` - Base node interface
+- `EntryNode` - Entry point node that receives tool arguments
+- `ExitNode` - Exit point node that returns final result
+- `McpNode` - MCP tool call node
+- `TransformNode` - JSONata transformation node
+- `SwitchNode` - JSON Logic routing node
+
+### 1.3 YAML Schema & Validation
+
+**File: `src/config/schema.ts`**
+
+Zod schemas implemented to validate YAML structure on load with clear error messages for invalid configurations.
+
+### 1.4 YAML Parser
+
+**File: `src/config/parser.ts`**
+
+YAML file parsing into structured configuration with validation against schema. Handles file I/O errors gracefully.
+
+### 1.5 Configuration Loader
+
+**File: `src/config/loader.ts`**
+
+Loads configuration from file path and returns parsed and validated configuration object.
+
+## Phase 2: Graph Structure & Representation
+
+### 2.1 Graph Data Structure
+
+**File: `src/graph/graph.ts`**
+
+Directed graph implementation from node definitions:
+- Node adjacency (edges) storage
+- Graph traversal utilities
+- Support for cycles
+
+### 2.2 Node Registry
+
+**Note:** Not implemented as separate file - functionality integrated into `Graph` class and `validator.ts`:
+- Node ID to node definition mapping
+- Node reference validation (tool references in entry/exit nodes, next, switch targets)
+- Orphaned node detection
+- Graph connectivity validation
+
+### 2.3 Graph Validator
+
+**File: `src/graph/validator.ts`**
+
+Graph structure validation:
+- All referenced nodes exist
+- All tools have exactly one entry and one exit node
+- Entry nodes are only referenced as tool entry points
+- Exit nodes are only referenced as tool exit points
+- Exit nodes are reachable from entry nodes
+- Detailed validation error messages
+
+## Phase 3: MCP Server Foundation
+
+### 3.1 MCP Server Setup
+
+**Note:** Not implemented as separate file - functionality integrated into `src/main.ts`:
+- MCP server initialization using `@modelcontextprotocol/sdk`
+- Stdio transport setup
+- Server lifecycle management
+
+### 3.2 Tool Registration
+
+**Note:** Not implemented as separate file - functionality integrated into `src/main.ts`:
+- Tool definitions converted to MCP tool schemas
+- Tools registered with MCP server
+- Tool names mapped to execution handlers
+
+### 3.3 Tool Execution Handler
+
+**Note:** Not implemented as separate file - functionality integrated into `src/main.ts`:
+- Tool invocation request handling
+- Tool argument extraction
+- Graph execution initiation at tool's entry node
+- Result return to MCP client
+
+## Phase 4: Expression Engines Integration
+
+### 4.1 JSONata Integration
+
+**File: `src/expressions/jsonata.ts`**
+
+JSONata library wrapper:
+- Expression evaluation with context data
+- Error handling
+- Support for JSONata references (e.g., `$.input.directory`)
+
+### 4.2 JSON Logic Integration
+
+**File: `src/expressions/json-logic.ts`**
+
+JSON Logic library wrapper:
+- Rule evaluation with context data
+- Boolean results for routing decisions
+- Error handling
+
+### 4.3 Expression Context
+
+**File: `src/execution/context.ts`**
+
+Expression evaluation context building:
+- Tool input arguments
+- Previous node outputs
+- Execution state
+- Data access for expressions
+
+**Note:** `src/expressions/context.ts` exists but functionality is primarily in `src/execution/context.ts`.
+
+## Phase 5: Node Execution
+
+### 5.1 Node Executor Base
+
+**Note:** Not implemented as separate base class - each executor is standalone with consistent execution pattern:
+- Pre-execution validation
+- Execute node logic
+- Post-execution processing
+- Determine next node(s)
+
+### 5.2 MCP Tool Node Executor
+
+**File: `src/execution/nodes/mcp-tool-executor.ts`**
+
+MCP tool node execution:
+- Pre-transform: Apply JSONata to format tool arguments
+- Call MCP tool using MCP client
+- Handle MCP call errors
+- Return output and next node
+
+### 5.3 Transform Node Executor
+
+**File: `src/execution/nodes/transform-executor.ts`**
+
+Transform node execution:
+- Apply JSONata transformation expression
+- Pass through data with transformation
+- Return transformed output and next node
+
+### 5.4 Switch Node Executor
+
+**File: `src/execution/nodes/switch-executor.ts`**
+
+Switch node execution:
+- Evaluate JSON Logic conditions in order
+- Select target node based on first matching condition
+- Handle default/fallback case (conditions without rules)
+- Return next node based on condition result
+
+### 5.5 Entry Node Executor
+
+**File: `src/execution/nodes/entry-executor.ts`**
+
+Entry node execution:
+- Receive tool arguments from MCP client
+- Initialize execution context with tool arguments
+- Make tool arguments available to subsequent nodes (e.g., `$.input.*`)
+- Pass execution to next node
+
+### 5.6 Exit Node Executor
+
+**File: `src/execution/nodes/exit-executor.ts`**
+
+Exit node execution:
+- Extract final result from execution context
+- Signal execution completion
+- Return final result to MCP tool caller
+
+## Phase 6: Graph Execution Engine
+
+### 6.1 Execution Context
+
+**File: `src/execution/context.ts`**
+
+Execution state management:
+- Current node tracking
+- Data context (tool inputs, node outputs)
+- Execution history
+- Error state
+- Data access for expressions
+
+### 6.2 Graph Executor
+
+**File: `src/execution/executor.ts`**
+
+Main graph execution orchestrator:
+- Custom sequential execution loop
+- Start at tool's entry node
+- Execute current node based on type (entry, mcp, transform, switch, exit)
+- Move to next node based on node's `next` field or switch routing
+- Continue until exit node is reached
+- Track execution history (node inputs/outputs)
+- Handle errors with context
+
+**Note:** The execution loop supports cycles (directed graphs with cycles), but infinite loops are prevented by the exit node check. Future loop node types can leverage this cycle support.
+
+### 6.3 Execution Flow
+
+**Note:** Not implemented as separate file - functionality integrated into `executor.ts`:
+- Node execution sequence coordination
+- Data flow between nodes
+- Branching (switch nodes)
+- Parallel execution support deferred
+
+## Phase 7: MCP Client for External Servers
+
+### 7.1 MCP Client Manager
+
+**File: `src/mcp/client-manager.ts`**
+
+MCP client connection management:
+- Connections to external MCP servers
+- Client connection caching
+- Connection lifecycle handling
+- Support for multiple concurrent servers
+
+### 7.2 MCP Client Factory
+
+**Note:** Not implemented as separate file - client creation functionality is in `client-manager.ts`:
+- MCP client creation for different server types
+- Stdio transport support
+- Client configuration
+
+### 7.3 Tool Call Handler
+
+**Note:** Not implemented as separate file - tool calling functionality is in `mcp-tool-executor.ts`:
+- `callTool` request execution to external MCP servers
+- Tool argument handling
+- Tool response processing
+- Error and timeout handling
+
+## Phase 8: Error Handling & Observability
+
+### 8.1 Error Types
+
+**Note:** Not implemented - using standard Error types:
+- Basic error handling works with standard Error
+- Custom error types (ConfigError, GraphError, ExecutionError, NodeError, McpError) would improve developer experience but are not required
+
+### 8.2 Error Handling
+
+**Note:** Basic error handling implemented throughout - centralized handler not implemented:
+- Error logging with context (via logger)
+- Error propagation through execution
+- Basic user-friendly error messages
+- Centralized error handler would be a nice-to-have enhancement
+
+### 8.3 Execution Logging
+
+**Note:** Basic logging implemented - structured execution logger not implemented:
+- Node execution events logged via basic logger
+- Structured logging would be valuable for debugging but basic logger works
+
+## Phase 9: Integration & Testing
+
+### 9.1 Main Entry Point
+
+**File: `src/main.ts`**
+
+Main entry point implementation:
+- Command-line argument parsing (config file path)
+- Configuration loading
+- Graph validation
+- MCP server startup
+- Tool registration
+- Tool execution handling
+- Graceful shutdown handling
+
+### 9.2 Integration Tests
+
+**Files: `tests/files.test.ts`, `tests/mcp-server.test.ts`, `tests/switch.test.ts`**
+
+Integration tests implemented:
+- Full execution flow with sample configs
+- Different node types tested
+- MCP client integration tested
+- Switch node conditional routing tested
+
+### 9.3 Sample Configurations
+
+**File: `examples/count_files.yaml`**
+
+Sample configuration implemented demonstrating:
+- Tool definition
+- Entry/exit nodes
+- MCP tool node
+- Transform node with JSONata
+
+## Phase 10: Polish & Documentation
+
+### 10.1 Code Documentation
+
+Basic JSDoc comments present on key functions. Comprehensive documentation would be a nice-to-have enhancement.
+
+### 10.2 Error Messages
+
+Basic error messages implemented. More helpful suggestions and context would improve developer experience.
+
+### 10.3 README Updates
+
+**File: `README.md`**
+
+README updated with:
+- Usage instructions
+- Installation from npm
+- MCP server configuration examples
+- Examples and configuration format documentation
+
+## Implementation Decisions
+
+1. **Custom Execution Engine**: A custom execution loop was implemented to provide full control over execution flow, enable observability (execution history), and support future debugging/introspection features.
+
+2. **Expression Evaluation**: All expressions (JSONata, JSON Logic) are evaluated with a consistent context that includes tool inputs and previous node outputs.
+
+3. **Data Flow**: Data flows through nodes as a JSON object that accumulates results. Each node can read from and write to this context.
+
+4. **Error Handling**: Errors at any node are caught, logged, and propagated. Basic error handling works; custom error types would be an enhancement.
+
+5. **MCP Client Management**: External MCP servers are managed as separate clients. The system maintains a registry of available MCP servers and their tools.
+
+6. **Code Organization**: Some planned separate files were integrated into existing files (e.g., server setup in main.ts, tool registration in main.ts). This works well and keeps the codebase simpler.
+
+## Future Considerations
+
+See `docs/future-introspection-debugging.md` for planned introspection and debugging features.
+
+Other future enhancements:
+- Visual editor/UX
+- Hot-reload of configuration
+- Loop node types (for, while, foreach)
+- Parallel node execution
+- Retry logic for failed nodes
+- Execution history persistence
+- Performance monitoring/metrics
+- OpenTelemetry integration
+- Custom error types
+- Structured execution logging
+- Centralized error handler
+