ruflo 3.5.2 → 3.5.4

package/src/ruvocal/docs/adr/ADR-033-RUVECTOR-RUFLO-MCP-INTEGRATION.md

@@ -0,0 +1,111 @@
+# ADR-033: RuVector + Ruflo MCP Tool Integration
+
+**Status:** Accepted
+**Date:** 2026-03-04
+**Context:** chat-ui-mcp MCP Bridge
+
+## Context
+
+The MCP bridge initially shipped with 3 built-in tools (search, web_research, system_guide). Users want access to the full ruvector (10 tools) and ruflo (205+ tools) ecosystems from within the HF Chat UI without running separate MCP servers.
+
+### Tool Inventory
+
+| Backend | Tools | Categories |
+|---------|-------|------------|
+| **ruvector** | 10 | Intelligence (hooks_stats, hooks_route, hooks_remember, hooks_recall, hooks_init, hooks_pretrain, hooks_build_agents, hooks_verify, hooks_doctor, hooks_export) |
+| **ruflo** | 205+ | Agent (7), Swarm (4), Memory (7), Config (6), Hooks (40+), Task (6), Session (5), Hive-mind (9), Workflow (9), Analyze (4), Progress (4), AIDefence (6), AgentDB (14+) |
+
+## Decision
+
+Integrate ruvector and ruflo as **stdio MCP child processes** spawned by the bridge, with tool calls proxied through the existing `/mcp` HTTP endpoint.
+
+### Architecture
+
+```
+┌─────────────────────────────────────────────────┐
+│  HF Chat UI (browser)                           │
+│  MCP_SERVERS: http://mcp-bridge:3001/mcp        │
+└─────────────────┬───────────────────────────────┘
+                  │  JSON-RPC 2.0 over HTTP
+                  ▼
+┌─────────────────────────────────────────────────┐
+│  MCP Bridge (Express)                           │
+│                                                 │
+│  ┌──────────────────┐  ┌─────────────────────┐ │
+│  │  Built-in Tools   │  │  StdioMcpClient    │ │
+│  │  • search         │  │  ┌───────────────┐ │ │
+│  │  • web_research   │  │  │ ruvector (10) │ │ │
+│  │  • system_guide   │  │  └───────────────┘ │ │
+│  └──────────────────┘  │  ┌───────────────┐ │ │
+│                         │  │ ruflo (205+)  │ │ │
+│                         │  └───────────────┘ │ │
+│                         └─────────────────────┘ │
+└─────────────────────────────────────────────────┘
+        ▲  stdin/stdout (JSON-RPC)  ▲
+        │                           │
+   npx ruvector mcp start    npx ruflo mcp start
+```
+
+### Key Design Decisions
+
+1. **Namespaced tool names**: External tools are prefixed with `{backend}__` (e.g., `ruvector__hooks_route`, `ruflo__agent_spawn`) to avoid name collisions with built-in tools.
+
+2. **Lazy startup**: Backends initialize after Express starts listening, so the bridge is immediately available for health checks. If a backend fails to start, built-in tools still work.
+
+3. **Environment toggle**: Each backend can be disabled via `ENABLE_RUVECTOR=false` or `ENABLE_RUFLO=false` for deployments that don't need all tools.
+
+4. **Graceful shutdown**: SIGTERM/SIGINT handlers kill child processes cleanly.
+
+5. **Timeout protection**: Each tool call has a 30s timeout. Backend initialization has a 15s timeout.
+
+## Implementation
+
+### StdioMcpClient
+
+A reusable client class that:
+- Spawns a child process with the MCP server command
+- Sends JSON-RPC messages over stdin, reads responses from stdout
+- Manages pending request map with UUID correlation IDs
+- Handles newline-delimited JSON protocol
+- Auto-discovers tools via `tools/list` on initialization
+
+### Tool Routing
+
+```
+tools/call request
+  → name starts with "{backend}__"?
+    → YES: strip prefix, route to StdioMcpClient.callTool()
+    → NO: route to built-in executeTool()
+```
+
+### Configuration
+
+```env
+# In docker-compose.yml or .env
+ENABLE_RUVECTOR=true    # default: true
+ENABLE_RUFLO=true       # default: true
+```
+
+## Consequences
+
+### Positive
+- 215+ tools available from HF Chat UI without separate MCP server management
+- Single `/mcp` endpoint — no client-side config changes needed
+- Built-in tools work even if backends fail to start
+- Namespacing prevents tool name collisions
+
+### Negative
+- Additional memory/CPU for child processes (~50MB each)
+- First request may be slow while npx resolves packages
+- Backend stderr goes to bridge logs (noisy)
+
+### Mitigations
+- Backends are optional (env toggle)
+- npx caches packages after first run
+- Startup is non-blocking
+
+## Related
+
+- [ADR-029: HuggingFace Chat UI Cloud Run](ADR-029-HUGGINGFACE-CHAT-UI-CLOUD-RUN.md)
+- [ADR-030: MCP Tool Gap Analysis](ADR-030-MCP-TOOL-GAP-ANALYSIS.md)
+- [ADR-032: RVF Private MCP Tunnel](ADR-032-RVF-PRIVATE-MCP-TUNNEL.md)

package/src/ruvocal/docs/adr/ADR-034-OPTIONAL-MCP-BACKENDS.md

@@ -0,0 +1,117 @@
+# ADR-034: Optional MCP Backends — Claude Code, Gemini, Codex
+
+**Status:** Accepted
+**Date:** 2026-03-05
+**Context:** chat-ui-mcp MCP Bridge
+
+## Context
+
+ADR-033 added ruvector (61 tools) and ruflo (215 tools) as default MCP backends. Users also want access to additional AI agent capabilities:
+
+- **Claude Code** — Anthropic's coding agent with file editing, bash execution, and code analysis tools
+- **Gemini MCP** — Google's Gemini model with conversation context management, multimodal capabilities
+- **OpenAI Codex** — OpenAI's coding agent for code generation and execution
+
+These require their own API keys and have different resource profiles, so they should be **opt-in** rather than default.
+
+## Decision
+
+Add three optional MCP backends that can be enabled via environment variables. Unlike ruvector/ruflo (enabled by default), these are **disabled by default** and require explicit API keys.
+
+### Backend Configuration
+
+| Backend | Env Toggle | API Key Required | Command | Default |
+|---------|-----------|-----------------|---------|---------|
+| ruvector | `ENABLE_RUVECTOR` | None | `npx ruvector mcp start` | **enabled** |
+| ruflo | `ENABLE_RUFLO` | None | `npx ruflo mcp start` | **enabled** |
+| Claude Code | `ENABLE_CLAUDE_CODE` | `ANTHROPIC_API_KEY` | `claude mcp serve` | disabled |
+| Gemini MCP | `ENABLE_GEMINI_MCP` | `GOOGLE_API_KEY` | `npx gemini-mcp-server` | disabled |
+| Codex | `ENABLE_CODEX` | `OPENAI_API_KEY` | `npx @openai/codex mcp serve` | disabled |
+
+### Architecture
+
+All backends use the same `StdioMcpClient` from ADR-033. Tools are namespaced by backend name:
+
+```
+ruvector__hooks_route      → ruvector MCP
+ruflo__agent_spawn         → ruflo MCP
+claude__Read               → Claude Code MCP
+gemini__chat               → Gemini MCP
+codex__execute             → Codex MCP
+```
+
+```
+┌───────────────────────────────────────────────────────┐
+│  MCP Bridge (/mcp)                                    │
+│                                                       │
+│  Built-in:  search, web_research, system_guide        │
+│                                                       │
+│  Default backends (always-on):                        │
+│  ┌─────────────┐  ┌──────────────┐                   │
+│  │ ruvector(61)│  │ ruflo (215) │                   │
+│  └─────────────┘  └──────────────┘                   │
+│                                                       │
+│  Optional backends (API key required):                │
+│  ┌──────────────┐  ┌───────────┐  ┌───────────────┐ │
+│  │ Claude Code  │  │ Gemini    │  │ OpenAI Codex │ │
+│  │ (opt-in)     │  │ (opt-in)  │  │ (opt-in)     │ │
+│  └──────────────┘  └───────────┘  └───────────────┘ │
+└───────────────────────────────────────────────────────┘
+```
+
+### Enabling Optional Backends
+
+```env
+# .env file
+ENABLE_CLAUDE_CODE=true
+ANTHROPIC_API_KEY=sk-ant-...
+
+ENABLE_GEMINI_MCP=true
+GOOGLE_API_KEY=AIzaSy...   # already set for Gemini models
+
+ENABLE_CODEX=true
+OPENAI_API_KEY=sk-...      # already set for OpenAI models
+```
+
+### Security Considerations
+
+1. **API keys stay server-side** — keys are only in the bridge container's env vars, never exposed to the browser
+2. **Optional by default** — backends that require API keys are disabled unless explicitly enabled
+3. **Graceful degradation** — if a backend fails to start (bad key, network error), built-in and other backends continue working
+4. **Namespace isolation** — tool name prefixing prevents cross-backend collisions
+
+### Resource Impact
+
+| Backend | Memory | CPU | Startup Time |
+|---------|--------|-----|-------------|
+| ruvector | ~30MB | Low | ~3s |
+| ruflo | ~50MB | Low | ~5s |
+| Claude Code | ~100MB | Medium | ~5s |
+| Gemini MCP | ~40MB | Low | ~4s |
+| Codex | ~80MB | Medium | ~5s |
+
+With all 5 backends enabled, the bridge container needs ~800MB memory.
+
+## Consequences
+
+### Positive
+- Users can access Claude, Gemini, and Codex capabilities directly from HF Chat UI
+- Single `/mcp` endpoint — no client-side config changes
+- Opt-in model keeps default resource usage low
+- API keys shared with the chat proxy (no additional secrets needed for Gemini/OpenAI)
+
+### Negative
+- Claude Code requires `@anthropic-ai/claude-code` installed (large package)
+- Each optional backend adds ~40-100MB memory when enabled
+- More child processes to manage in the container
+
+### Mitigations
+- Backends pre-installed in Docker image for fast startup
+- Disabled by default — only started when explicitly enabled
+- Health endpoint reports backend status for debugging
+
+## Related
+
+- [ADR-033: RuVector + Ruflo MCP Integration](ADR-033-RUVECTOR-RUFLO-MCP-INTEGRATION.md)
+- [ADR-032: RVF Private MCP Tunnel](ADR-032-RVF-PRIVATE-MCP-TUNNEL.md)
+- [ADR-029: HuggingFace Chat UI Cloud Run](ADR-029-HUGGINGFACE-CHAT-UI-CLOUD-RUN.md)

package/src/ruvocal/docs/adr/ADR-035-MCP-TOOL-GROUPS.md

@@ -0,0 +1,186 @@
+# ADR-035: MCP Tool Groups — Modular Tool Organization
+
+**Status:** Accepted
+**Date:** 2026-03-05
+**Supersedes:** ADR-033, ADR-034
+
+## Context
+
+The MCP bridge grew to 331+ tools from multiple backends (ruvector, ruflo, agentic-flow, Claude Code, Gemini, Codex). Exposing all tools simultaneously caused:
+
+1. **Context flooding** — AI models struggle to select the right tool from 300+ options
+2. **Startup overhead** — loading all backends when only a subset is needed
+3. **No discoverability** — the AI had no structured way to learn about available capabilities
+
+## Decision
+
+Reorganize all tools into **12 logical groups** that can be independently enabled/disabled via `MCP_GROUP_*` environment variables. Add a built-in `guidance` tool that provides structured instructions to the AI about available capabilities.
+
+### Tool Groups
+
+| Group | Source | Tools | Default | Env Var |
+|-------|--------|-------|---------|---------|
+| **core** | built-in | search, web_research, guidance | always on | — |
+| **intelligence** | ruvector | ~10 | enabled | `MCP_GROUP_INTELLIGENCE` |
+| **agents** | ruflo | ~50 | enabled | `MCP_GROUP_AGENTS` |
+| **memory** | ruflo | ~25 | enabled | `MCP_GROUP_MEMORY` |
+| **devtools** | ruflo | ~60 | enabled | `MCP_GROUP_DEVTOOLS` |
+| **security** | ruflo | ~25 | disabled | `MCP_GROUP_SECURITY` |
+| **browser** | ruflo | ~23 | disabled | `MCP_GROUP_BROWSER` |
+| **neural** | ruflo | ~20 | disabled | `MCP_GROUP_NEURAL` |
+| **agentic-flow** | agentic-flow@alpha | 15 | disabled | `MCP_GROUP_AGENTIC_FLOW` |
+| **claude-code** | claude mcp serve | varies | disabled | `MCP_GROUP_CLAUDE_CODE` |
+| **gemini** | gemini-mcp-server | varies | disabled | `MCP_GROUP_GEMINI` |
+| **codex** | @openai/codex | varies | disabled | `MCP_GROUP_CODEX` |
+
+### Architecture
+
+```
+┌─────────────────────────────────────────────────────────┐
+│  HF Chat UI → /mcp                                     │
+└─────────────┬───────────────────────────────────────────┘
+              ▼
+┌─────────────────────────────────────────────────────────┐
+│  MCP Bridge v2.0.0                                      │
+│                                                         │
+│  ┌─────────────────────────────────────────────────┐   │
+│  │  TOOL GROUP FILTER                               │   │
+│  │  MCP_GROUP_INTELLIGENCE=true  → include          │   │
+│  │  MCP_GROUP_AGENTS=true        → include          │   │
+│  │  MCP_GROUP_BROWSER=false      → exclude          │   │
+│  │  MCP_GROUP_NEURAL=false       → exclude          │   │
+│  └─────────────────────────────────────────────────┘   │
+│         ▼                    ▼                ▼         │
+│  ┌──────────┐  ┌──────────────┐  ┌─────────────────┐  │
+│  │ ruvector │  │    ruflo     │  │ agentic-flow    │  │
+│  │ (stdio)  │  │   (stdio)   │  │    (stdio)      │  │
+│  └──────────┘  └──────────────┘  └─────────────────┘  │
+│                                                         │
+│  Optional (disabled by default):                        │
+│  ┌──────────┐  ┌──────────┐  ┌──────────┐             │
+│  │  Claude  │  │  Gemini  │  │  Codex   │             │
+│  └──────────┘  └──────────┘  └──────────┘             │
+└─────────────────────────────────────────────────────────┘
+```
+
+### Group Filtering
+
+Tools from external backends are filtered by matching their original tool name against group prefix patterns:
+
+```javascript
+// Group definition
+agents: {
+  source: "ruflo",
+  prefixes: ["agent_", "swarm_", "task_", "session_", "hive-mind_", "workflow_", "coordination_"],
+}
+
+// ruflo tool "agent_spawn" → matches "agent_" prefix → included if agents group enabled
+// ruflo tool "browser_open" → matches "browser_" prefix → only if browser group enabled
+```
+
+A backend is only started if at least one of its groups is enabled. This means disabling all ruflo groups prevents the ruflo process from spawning entirely.
+
+### Guidance Tool
+
+The `guidance` tool replaces the old `system_guide`. It provides structured, AI-optimized instructions:
+
+```
+guidance(topic="overview")     → capabilities summary + decision guide
+guidance(topic="groups")       → table of all groups with status
+guidance(topic="agents")       → detailed usage for the agents group
+guidance(topic="tool", tool_name="ruflo__memory_search") → specific tool docs
+```
+
+The system prompt instructs the AI to call `guidance` when:
+- Unsure which tool to use
+- User asks "what can you do?"
+- Needs to learn a specific tool group before using it
+
+### Agentic-Flow Integration
+
+`agentic-flow@alpha` (npm package) provides 15 tools:
+
+| Tool | Description |
+|------|-------------|
+| `agentic_flow_agent` | Execute any of 66+ specialized agents |
+| `agentic_flow_list_agents` | List available agent types |
+| `agentic_flow_create_agent` | Create custom agents |
+| `agentic_flow_list_all_agents` | List with sources |
+| `agentic_flow_agent_info` | Get agent details |
+| `agentic_flow_check_conflicts` | Agent conflict detection |
+| `agentic_flow_optimize_model` | Auto-select best model |
+| `agent_booster_edit_file` | 352x faster code editing |
+| `agent_booster_batch_edit` | Multi-file refactoring |
+| `agent_booster_parse_markdown` | LLM output parsing |
+| `agentdb_stats` | Database statistics |
+| `agentdb_pattern_store` | Store reasoning patterns |
+| `agentdb_pattern_search` | Search similar patterns |
+| `agentdb_pattern_stats` | Pattern analytics |
+| `agentdb_clear_cache` | Clear query cache |
+
+## Configuration Examples
+
+### Minimal (research assistant)
+```env
+MCP_GROUP_INTELLIGENCE=false
+MCP_GROUP_AGENTS=false
+MCP_GROUP_MEMORY=false
+MCP_GROUP_DEVTOOLS=false
+# Only core tools: search, web_research, guidance
+```
+
+### Developer workstation
+```env
+MCP_GROUP_INTELLIGENCE=true
+MCP_GROUP_AGENTS=true
+MCP_GROUP_MEMORY=true
+MCP_GROUP_DEVTOOLS=true
+MCP_GROUP_AGENTIC_FLOW=true   # agent execution + boosted editing
+```
+
+### Full capabilities
+```env
+MCP_GROUP_INTELLIGENCE=true
+MCP_GROUP_AGENTS=true
+MCP_GROUP_MEMORY=true
+MCP_GROUP_DEVTOOLS=true
+MCP_GROUP_SECURITY=true
+MCP_GROUP_BROWSER=true
+MCP_GROUP_NEURAL=true
+MCP_GROUP_AGENTIC_FLOW=true
+MCP_GROUP_CLAUDE_CODE=true
+MCP_GROUP_GEMINI=true
+MCP_GROUP_CODEX=true
+ANTHROPIC_API_KEY=sk-ant-...
+```
+
+## API Endpoints
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/health` | GET | System health with group status |
+| `/groups` | GET | Detailed group info with tool counts |
+| `/models` | GET | Available LLM models |
+| `/mcp` | POST | MCP JSON-RPC (tools/list, tools/call) |
+
+## Consequences
+
+### Positive
+- AI sees only relevant tools (20-50 instead of 300+), improving tool selection accuracy
+- Unused backends don't start, saving memory and CPU
+- `guidance` tool provides structured discoverability
+- Groups can be mixed and matched per deployment
+- New backends/groups can be added without touching existing code
+
+### Negative
+- Some tools appear in multiple potential groups (e.g., ruflo `hooks_*` in both intelligence and devtools) — resolved by prefix matching
+- Group boundaries are somewhat arbitrary for the ruflo "Uncategorized" tools
+
+### Mitigations
+- `guidance` tool helps AI navigate regardless of how tools are grouped
+- `/groups` endpoint lets operators inspect what's actually active
+
+## Related
+
+- [ADR-029: HuggingFace Chat UI Cloud Run](ADR-029-HUGGINGFACE-CHAT-UI-CLOUD-RUN.md)
+- [ADR-032: RVF Private MCP Tunnel](ADR-032-RVF-PRIVATE-MCP-TUNNEL.md)