sofia-cli 0.1.1

package/specs/003-mcp-transport-integration/research.md

@@ -0,0 +1,395 @@
+# Research: MCP Transport Integration
+
+**Feature ID**: 003-mcp-transport-integration  
+**Date**: 2026-03-01  
+**Status**: Complete
+
+---
+
+## Topic 1: GitHub Copilot SDK MCP Support (FR-019)
+
+### Question
+
+Does `@github/copilot-sdk` v0.1.28 provide native MCP transport — tool dispatch, server lifecycle management, stdio/HTTP protocol handling, and authentication? Where must we build custom transport?
+
+### Findings
+
+**SDK scope (v0.1.28) — UPDATED**: The Copilot SDK provides **native MCP server management** via `SessionConfig.mcpServers`. When `mcpServers` is passed to `CopilotClient.createSession()`, the SDK handles the full lifecycle: spawning stdio subprocesses, connecting to HTTP endpoints, JSON-RPC framing, and tool dispatch during LLM conversations. The SDK exposes `MCPLocalServerConfig` (type `"local"` | `"stdio"`, with `command`, `args`, `env?`, `cwd?`, `tools`, `timeout?`) and `MCPRemoteServerConfig` (type `"http"` | `"sse"`, with `url`, `headers?`, `tools`, `timeout?`). These types map directly to `.vscode/mcp.json` server entries.
+
+**However**, the SDK's `executeToolCall()` method on `CopilotClient` is **private** — there is no public API to invoke a tool programmatically outside of an LLM conversation turn. This means:
+
+- **LLM-initiated tool calls** (e.g., web search during conversation, discovery enrichment) can leverage SDK-managed MCP servers. The SDK spawns, connects, and dispatches tool calls as needed — no custom transport required.
+- **Programmatic adapter calls** (GitHub `createRepository`, Context7 `resolve-library-id`, Azure `documentation`) cannot use the SDK. These calls are made deterministically by application code, not by the LLM. They require our custom `McpTransport` layer.
+
+**Dual-path architecture**:
+
+1. **SDK-managed path (LLM conversations)**: Pass `.vscode/mcp.json` config as `mcpServers` to `sdkClient.createSession()`. The SDK handles server lifecycle, JSON-RPC, and tool dispatch during `sendAndWait()` calls. Tools are available to the LLM without additional sofIA code.
+2. **Custom transport path (programmatic adapter calls)**: `McpManager.callTool()` uses our `StdioMcpTransport` / `HttpMcpTransport` for deterministic tool invocations by adapters (GitHub, Context7, Azure). These bypass the LLM entirely.
+
+**SDK value in this feature**: The SDK is useful for both (a) the web search case (`web.search` tool) — already registered via `WEB_SEARCH_TOOL_DEFINITION` — and (b) making any MCP server's tools available to the LLM during conversation turns. For direct MCP tool calls made by adapters (GitHub adapter, Context7 enricher), the SDK is bypassed — adapters call `McpManager.callTool()` directly without going through the LLM conversation.
+
+**⚠️ Dual-lifecycle limitation**: If the same MCP server (e.g., `context7`) is used both by the SDK (LLM conversation) and by a custom transport (programmatic adapter call), two separate subprocesses will be spawned. This is acceptable for Feature 003 scope but should be revisited if subprocess resource usage becomes a concern in later features.
+
+### Decision
+
+**Dual-path approach**:
+
+1. **Wire SDK-managed MCP** — Pass `.vscode/mcp.json` config (converted to `MCPServerConfig` format) to `sdkClient.createSession({ mcpServers })` so the LLM can invoke MCP tools during conversation turns. This requires updating `copilotClient.ts` to accept and forward `mcpServers`.
+2. **Build custom MCP transport in `src/mcp/mcpTransport.ts`** — For the programmatic adapter path only (GitHub, Context7, Azure). The SDK's `executeToolCall` is private, so direct tool invocation from application code requires custom transport.
+
+**Rationale**: Leveraging the SDK's native `mcpServers` support reduces the amount of custom protocol code needed for LLM conversation paths. The custom transport is still necessary for deterministic adapter calls that bypass the LLM. This dual approach aligns with the SDK's design intent.
+
+**Alternatives considered**:
+
+- Use the SDK tool-call loop for all MCP calls — rejected because it requires the LLM to initiate every tool call, making GitHub repo creation dependent on LLM decisions rather than deterministic code.
+- Use only custom transport for everything — rejected because it ignores the SDK's native MCP support, duplicating server lifecycle management the SDK already provides for LLM conversations.
+- Use a third-party MCP client library (`@modelcontextprotocol/sdk`) — viable but adds a dependency. The MCP client protocol (JSON-RPC 2.0 over stdio/HTTP) is simple enough to implement directly for the adapter path. Revisit in a later feature if the protocol surface grows.
+
+---
+
+## Topic 2: MCP Transport Protocol (stdio vs HTTP)
+
+### Question
+
+What protocol framing does each MCP server type require? How do stdio servers (Context7, Azure, WorkIQ, Playwright) communicate, and how do HTTP servers (GitHub MCP, Microsoft Docs MCP) differ?
+
+### Findings
+
+**MCP Protocol**: JSON-RPC 2.0. Every request is a JSON object:
+
+```json
+{
+  "jsonrpc": "2.0",
+  "id": 1,
+  "method": "tools/call",
+  "params": { "name": "toolName", "arguments": {} }
+}
+```
+
+Every response is:
+
+```json
+{ "jsonrpc": "2.0", "id": 1, "result": { "content": [{ "type": "text", "text": "..." }] } }
+```
+
+Errors use the standard JSON-RPC error envelope.
+
+**Stdio servers** (Context7: `npx @upstash/context7-mcp`, Azure: `npx @azure/mcp server start`, WorkIQ: `npx @microsoft/workiq mcp`):
+
+- Spawned as child processes (`child_process.spawn` with `stdio: ['pipe','pipe','pipe']`)
+- JSON-RPC messages delimited by newlines over stdin/stdout
+- One subprocess per server, kept alive for the session duration
+- Initialization: send `initialize` request first, receive `initialized` notification
+- Auth: inherits environment variables from the parent process (e.g., `GITHUB_TOKEN`, `AZURE_SUBSCRIPTION_ID`)
+
+**HTTP servers** (GitHub MCP: `https://api.githubcopilot.com/mcp/`, Microsoft Docs: `https://learn.microsoft.com/api/mcp`):
+
+- Standard HTTPS POST to the server URL with `Content-Type: application/json`
+- Response is JSON-RPC response body
+- Auth: HTTP Authorization header — GitHub MCP uses the user's Copilot token (extracted from the SDK session context or `GITHUB_TOKEN` env var)
+- No persistent connection needed; each tool call is a stateless HTTP request
+
+### Decision
+
+**Implement two transport strategies**:
+
+1. `StdioMcpTransport`: spawns subprocess, maintains persistent stdin/stdout pipe, sequences requests with pending-request map keyed by JSON-RPC id.
+2. `HttpMcpTransport`: wraps native `fetch()`, adds `Authorization` header from env `GITHUB_TOKEN`, sets timeout via `AbortController`.
+
+Both implement a common `McpTransport` interface:
+
+```typescript
+interface McpTransport {
+  callTool(
+    toolName: string,
+    args: Record<string, unknown>,
+    timeoutMs: number,
+  ): Promise<ToolCallResponse>;
+  isConnected(): boolean;
+  disconnect(): Promise<void>;
+}
+```
+
+**Rationale**: Clean abstraction isolates protocol details from `McpManager`, making both easily testable with mock transports.
+
+**Alternatives considered**:
+
+- Single class with type switch — rejected because it conflates two fundamentally different I/O models.
+- Streaming responses (SSE) — deferred; none of the current MCP servers require streaming at the tool-call level.
+
+---
+
+## Topic 3: Authentication Model Per Transport
+
+### Question
+
+How does each MCP server authenticate tool calls? What credentials are needed and how are they passed?
+
+### Findings
+
+| Server              | Transport | Auth Mechanism                                                                                                 | Source                                              |
+| ------------------- | --------- | -------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- |
+| `github`            | HTTP      | `Authorization: Bearer <token>`                                                                                | `GITHUB_TOKEN` env var (set by Copilot environment) |
+| `context7`          | stdio     | None required for public package                                                                               | npx subprocess, no auth needed                      |
+| `azure`             | stdio     | Azure identity from env (`AZURE_SUBSCRIPTION_ID`, `AZURE_TENANT_ID`, `AZURE_CLIENT_ID`/DefaultAzureCredential) | Subprocess inherits env                             |
+| `workiq`            | stdio     | Microsoft 365 OAuth — WorkIQ handles its own auth flow when launched                                           | Subprocess prompts user or reads cached token       |
+| `microsoftdocs/mcp` | HTTP      | None (public API)                                                                                              | No auth header needed                               |
+
+**Key finding for WorkIQ**: WorkIQ's own auth flow means sofIA does not need to implement OAuth — it only needs to spawn the subprocess. However, WorkIQ will prompt the user to authenticate on first use. This is handled transparently by the subprocess.
+
+**Key finding for GitHub HTTP transport**: The `GITHUB_TOKEN` environment variable is the standard Copilot-set credential. The `HttpMcpTransport` reads it at call time (not startup) to avoid storing it in memory.
+
+### Decision
+
+Auth is **transport-level, not application-level**:
+
+- `HttpMcpTransport` reads `process.env.GITHUB_TOKEN` per call, adds it as Bearer token.
+- `StdioMcpTransport` passes the parent's `process.env` to the subprocess unchanged.
+- No custom auth abstraction is needed. WorkIQ's own flow covers M365 auth.
+
+---
+
+## Topic 4: Retry Policy
+
+### Question
+
+Which error types are transient (retryable) vs permanent (must not retry)?
+
+### Findings
+
+From the spec (FR-004a) and error classification in `classifyMcpError()`:
+
+| Error Class            | Retryable     | Examples                                            |
+| ---------------------- | ------------- | --------------------------------------------------- |
+| `connection-refused`   | ✅ Yes        | MCP subprocess not yet ready, temporary port in use |
+| `timeout`              | ✅ Yes        | Server temporarily slow, network hiccup             |
+| `dns-failure`          | ✅ Yes (once) | Transient DNS issues                                |
+| `auth-failure`         | ❌ No         | Invalid token, expired credentials, 401/403 HTTP    |
+| `unknown`              | ❌ No         | Malformed JSON response, logic errors               |
+| Validation error (Zod) | ❌ No         | Bad args, schema mismatch                           |
+
+**Backoff**: 1 second initial delay (jittered ±20%), exponential factor 2x, maximum 1 retry (as per spec — one automatic retry).
+
+### Decision
+
+**`retryPolicy.ts`** exports a `withRetry<T>(fn, options)` helper that wraps any async call:
+
+- Retries once on transient errors after `initialDelayMs` (default 1000ms) with ±20% jitter.
+- Does not retry on auth failures, validation errors, or unknown errors.
+- On retry, logs a `warn` entry with server name, tool name, attempt number, and delay.
+
+---
+
+## Topic 5: `pushFiles` Empty Content Bug (GAP-003)
+
+### Question
+
+Where exactly does `ralphLoop.ts` send empty file content, and what is the minimal fix?
+
+### Findings
+
+In `ralphLoop.ts` lines 527–548, the current code already reads file content from disk using `readFile(resolve(outputDir, f), 'utf-8')` in the success-path push. The `content: ''` fallback at line 540 is reached only when `readFile` throws (file unreadable). The **actual bug** is that this same pattern is NOT used in the first-iteration scaffold push — only `applyResult.writtenFiles` paths are pushed after iterations 2+.
+
+Reviewing the code more carefully: the current implementation at lines 527–548 does read file content. The GAP-003 bug noted in `specs/003-next-spec-gaps.md` was based on an earlier version. The current code's `content: ''` at line 540 is a **fallback for unreadable files** (correct behavior — skip unreadable rather than push garbage). However, the first scaffold push (before iteration 2) is missing — files created by `PocScaffolder` are never pushed to GitHub.
+
+**Fix needed**: After the initial scaffold completes and a GitHub repo is created, push the scaffold files once (all `scaffoldResult.createdFiles`) with their actual on-disk content. The per-iteration push in the main loop already reads content correctly.
+
+### Decision
+
+Add a post-scaffold push in `RalphLoop.run()` immediately after the npm install step, reading all scaffold file contents from disk (same `readFile` pattern already used in the iteration push block).
+
+---
+
+## Topic 6: Discovery Phase Enrichment Architecture (GAP-005)
+
+### Question
+
+Where in the discovery phase flow should web search and WorkIQ enrichment be inserted? How is `DiscoveryEnrichment` stored in the session?
+
+### Findings
+
+From `src/phases/phaseHandlers.ts` and the spec (FR-014 through FR-018):
+
+- Step 1 of the discovery phase collects company and team information and stores it in `DiscoveryState`.
+- After this collection, the enricher should be triggered.
+- The session schema (`src/shared/schemas/session.ts`) has a `DiscoveryState` object that currently does not have an enrichment field.
+
+**Flow design**:
+
+1. User completes Step 1 input.
+2. `discoveryEnricher.ts` is invoked with the step 1 summary.
+3. It calls `web.search` via `webSearch.ts` (already implemented) for company/competitor/industry queries.
+4. It optionally calls WorkIQ after showing a permission prompt.
+5. Results are stored in `session.discovery.enrichment: DiscoveryEnrichment`.
+6. The enrichment is referenced in subsequent phase prompts (ideation, planning).
+
+**WorkIQ permission**: Implemented as an `@inquirer/prompts` `confirm` prompt before any WorkIQ call. If the user declines, or WorkIQ subprocess is not available, enrichment is skipped silently.
+
+### Decision
+
+New module `src/phases/discoveryEnricher.ts` with a `DiscoveryEnricher` class that has:
+
+- `enrichFromWebSearch(companySummary, mcpManager): Promise<Partial<DiscoveryEnrichment>>`
+- `enrichFromWorkIQ(companySummary, mcpManager, io): Promise<Partial<DiscoveryEnrichment>>`
+- `enrich(companySummary, mcpManager, io): Promise<DiscoveryEnrichment>` — orchestrates both, handles graceful degradation.
+
+`DiscoveryEnrichment` is added to the Zod session schema as an optional field on `DiscoveryState`.
+
+---
+
+## Topic 7: Copilot SDK Agent Architecture Alignment (FR-020)
+
+### Question
+
+Do the current agent definitions (discovery, ideation, design, select, plan, develop) need structural changes to align with Copilot SDK v0.1.28 patterns?
+
+### Findings
+
+The Copilot SDK v0.1.28 agent model:
+
+- An "agent" is a function/class that creates sessions via `CopilotClient.createSession(options)` where `options.systemPrompt` is the agent's identity and `options.tools` is the tool set.
+- Tools are declared as `ToolDefinition[]`; there is no formal "agent registry" API in v0.1.28.
+- The SDK dispatches `tool_call` events which the host handles in the conversation loop.
+
+**Current sofIA agents**: Each phase (discovery, ideation, design, select, plan, develop) uses `createFakeCopilotClient` in tests and the real SDK client in production. They declare tools via `SessionOptions.tools`. This matches the SDK's expected pattern exactly.
+
+**No structural misalignment found**. The current architecture correctly uses:
+
+- `CopilotClient.createSession()` → one session per phase turn.
+- `ToolDefinition` objects for capability declaration.
+- Conversation loop event handling for tool dispatch.
+
+**What FR-020 means in practice**: Ensure that when `callTool()` is implemented, direct adapter calls (GitHub, Context7, Azure) do NOT go through the LLM session — they are application-layer calls. This is already the design. The "alignment" is confirming the SDK does not provide a competing pattern that we should use instead.
+
+### Decision
+
+**No agent refactoring needed**. The existing architecture is already SDK-aligned. FR-020 is satisfied by documenting this finding in `research.md` (this document) and ensuring `McpManager.callTool()` is an application-layer call, not routed through LLM sessions.
+
+---
+
+## Topic 8: SDK Hooks, Events, and CLI Transparency (FR-021, FR-022, FR-024)
+
+### Question
+
+How should sofIA use the Copilot SDK's hooks and event system to provide real-time visibility into tool activity, errors, and usage — satisfying Constitution Principle VIII (CLI-First UX & Transparency)?
+
+### Findings
+
+The Copilot SDK v0.1.28 provides two complementary mechanisms for runtime visibility:
+
+**Hooks** (via `SessionConfig.hooks`):
+
+Six lifecycle hooks are available:
+
+- `onPreToolUse(toolName, toolArgs, context)` — fired before every tool call; returns `{ permissionDecision, reason }` to allow/deny
+- `onPostToolUse(toolResult, context)` — fired after every tool call; can modify or log results
+- `onUserPromptSubmitted(prompt, context)` — modify user prompts before processing
+- `onSessionStart(context)` — add additional context at session start
+- `onSessionEnd(context)` — cleanup/analytics
+- `onErrorOccurred(error, context)` — custom error handling for LLM-path errors
+
+**Events** (via `session.on()`/`session.once()`):
+
+40+ event types are available, including:
+
+- `assistant.usage` — token usage per turn (input/output tokens)
+- Streaming delta events for real-time content display
+- Tool call lifecycle events
+
+**Key insight for CLI transparency**: `onPreToolUse` and `onPostToolUse` are the standard mechanism to implement Constitution Principle VIII's requirement that _"Users MUST always see the current execution state."_ Currently, sofIA's spinner shows phase-level activity but does NOT show individual MCP tool calls being made during LLM conversation turns. The SDK hooks are the native way to emit this visibility.
+
+**`onErrorOccurred` for LLM-path errors**: FR-004's `classifyMcpError()` only covers the custom transport path (adapter calls). Errors during SDK-managed tool calls (LLM conversation path) are handled by the SDK internally. The `onErrorOccurred` hook allows sofIA to log, surface, or recover from these errors — complementing the custom transport error handling.
+
+**`onPermissionRequest` for tool approval**: The SDK provides an `onPermissionRequest` handler that implements deny-by-default tool approval. This was evaluated as an alternative to `io.prompt()` for WorkIQ consent (FR-016). Decision: defer in favor of the existing `io.prompt()` pattern, which is consistent with other interactive prompts in the discovery phase and supports custom consent UX.
+
+**`assistant.usage` for token tracking**: Subscribing to the `assistant.usage` event provides per-turn token counts. This can be logged at `debug` level and optionally displayed in the spinner for transparency during long-running sessions.
+
+### Decision
+
+**Wire SDK hooks for tool-call visibility**:
+
+1. Add `hooks` support to `SessionOptions` in `copilotClient.ts`.
+2. Wire `onPreToolUse` to emit a `tool:start` activity event (tool name) to the CLI spinner.
+3. Wire `onPostToolUse` to emit a `tool:end` activity event (tool name, duration) to the CLI spinner.
+4. Wire `onErrorOccurred` to log SDK-path errors at `warn` level via the existing pino logger.
+5. Subscribe to `assistant.usage` events and log token usage at `debug` level.
+
+**Defer**: `onPermissionRequest` (use `io.prompt()` instead for WorkIQ consent), `onUserPromptSubmitted` (no current use case), `onSessionStart`/`onSessionEnd` (no current use case beyond what's already handled).
+
+**Rationale**: Hooks are the SDK-native mechanism for the transparency that Constitution Principle VIII requires. Without them, MCP tool calls during LLM conversation turns are invisible to the user. The implementation is low-effort: forward hooks to `createSession()`, emit events to the existing spinner infrastructure in `src/shared/events.ts`.
+
+---
+
+## Topic 9: SDK Advanced Session Features — infiniteSessions, customAgents, skillDirectories (FR-023)
+
+### Question
+
+Do the SDK's `infiniteSessions`, `customAgents`, and `skillDirectories` features offer advantages over sofIA's current implementation patterns?
+
+### Findings
+
+**`infiniteSessions` config**:
+
+- Controls context window management for long-running sessions.
+- `backgroundCompactionThreshold` (default 0.7): triggers background context compaction when usage exceeds this ratio.
+- `bufferExhaustionThreshold` (default 0.9): forces compaction to prevent context overflow.
+- **Direct relevance**: The Ralph Loop runs extended multi-iteration conversations (up to `maxIterations` turns with code generation, test output, and enrichment context). Without `infiniteSessions`, long runs risk silently truncating conversation history, losing important context about failing tests or previous code changes.
+- **Current gap**: Neither spec nor tasks configure `infiniteSessions`. The Ralph Loop could hit context limits on iteration 8+ with verbose test output.
+
+**`customAgents` config**:
+
+- Allows defining multiple agent personas within a single session.
+- Each agent has its own system prompt, tools, and capabilities.
+- The SDK handles agent switching within the session.
+- **sofIA's current pattern**: Creates a new session per phase via `createSession()`. Each phase has its own system prompt and tools.
+- **Evaluation**: `customAgents` would allow all phases to share a single session, but sofIA intentionally isolates phases with separate sessions for:
+  - Clean context boundaries between workshop phases
+  - Independent session history per phase
+  - Ability to checkpoint/resume individual phases
+- **Conclusion**: The per-phase session pattern is deliberate and offers advantages that `customAgents` would sacrifice. No change needed.
+
+**`skillDirectories` config**:
+
+- Skills are named directories containing `SKILL.md` files (markdown with optional YAML frontmatter).
+- Content is injected into the session context.
+- Can disable specific skills via `disabledSkills`.
+- **sofIA's current pattern**: `promptLoader.ts` loads prompts from `src/prompts/` as markdown files, injects them as system prompts via `SessionOptions.systemMessage`.
+- **Evaluation**: `skillDirectories` could replace `promptLoader.ts` for phase-specific prompts, but:
+  - `promptLoader.ts` already works correctly and is tested.
+  - Skills are additive context injection, not primary system prompts — semantically different.
+  - Migration would add complexity without clear benefit.
+- **Conclusion**: Keep `promptLoader.ts`. Skills could be used for supplementary context (e.g., workshop materials, card decks) in future features.
+
+**`resumeSession(sessionId)` + session persistence**:
+
+- Sessions persist at `~/.copilot/session-state/{sessionId}/` with checkpoints, plan.md, files.
+- `resumeSession()` restores conversation history, tool call results, agent planning state.
+- **Current "Out of Scope" deferral**: "Resume/checkpoint for `sofia dev`" (GAP-006 P2) was deferred assuming significant implementation effort.
+- **SDK reality**: The SDK handles persistence natively — sofIA only needs to pass a structured `sessionId` and call `resumeSession()`. Implementation complexity is much lower than originally assessed.
+- **Conclusion**: Keep deferred (different feature scope) but note reduced complexity in spec's Out of Scope section.
+
+### Decision
+
+| Feature            | Decision                                                        | Rationale                                                                            |
+| ------------------ | --------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
+| `infiniteSessions` | **Wire for Ralph Loop sessions**                                | Prevents context window exhaustion in extended iterations; low implementation effort |
+| `customAgents`     | **Defer — current per-phase sessions are deliberate**           | Phase isolation provides clean context boundaries and independent checkpointing      |
+| `skillDirectories` | **Defer — `promptLoader.ts` is sufficient**                     | Current approach works; skills could supplement in future features                   |
+| `resumeSession`    | **Defer (different feature scope) but note reduced complexity** | SDK makes this nearly trivial; updated Out of Scope note in spec.md                  |
+
+---
+
+## Summary of Decisions
+
+| Topic                | Decision                                                                                                                                                                                                  |
+| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| SDK MCP support      | SDK provides native `mcpServers` in `createSession()` for LLM conversations; build custom `mcpTransport.ts` only for programmatic adapter calls (GitHub, Context7, Azure)                                 |
+| stdio transport      | `child_process.spawn` + JSON-RPC 2.0 newline-delimited over stdin/stdout                                                                                                                                  |
+| HTTP transport       | Native `fetch()` + `AbortController` timeout + `Authorization: Bearer`                                                                                                                                    |
+| Auth model           | Transport-level: env var for HTTP, subprocess env inheritance for stdio                                                                                                                                   |
+| Retry policy         | 1 retry max, transient errors only, 1s base delay ±20% jitter                                                                                                                                             |
+| pushFiles bug        | Add post-scaffold push; per-iteration push already correct                                                                                                                                                |
+| Discovery enrichment | New `discoveryEnricher.ts`; `DiscoveryEnrichment` in session schema                                                                                                                                       |
+| Agent alignment      | No refactoring needed; current SDK usage is correct                                                                                                                                                       |
+| SDK hooks & events   | Wire `onPreToolUse`/`onPostToolUse` for CLI spinner visibility; `onErrorOccurred` for LLM-path errors; `assistant.usage` for token tracking                                                               |
+| SDK session features | `infiniteSessions` wired for Ralph Loop; `customAgents` deferred (per-phase sessions deliberate); `skillDirectories` deferred (`promptLoader.ts` sufficient); session persistence noted as low-complexity |

package/specs/003-mcp-transport-integration/spec.md

@@ -0,0 +1,234 @@
+# Feature Specification: MCP Transport Integration
+
+**Feature Branch**: `003-mcp-transport-integration`  
+**Created**: 2026-03-01  
+**Status**: Complete  
+**Completed**: 2026-03-02  
+**Upstream Dependency**: specs/002-poc-generation/spec.md (Ralph Loop, GitHub adapter, context enricher)  
+**Input**: User description: "Implement real MCP tool invocation layer connecting McpManager to actual MCP server transports, enabling GitHub repository creation, Context7 documentation lookup, Azure architecture guidance, and web search capabilities to function in production"
+
+## Overview
+
+Feature 002 built the Ralph Loop iteration engine and all its MCP-powered components — GitHub adapter, Context7 enricher, Azure enricher, web search enricher — but every MCP tool call is currently a stub that either returns fake data or throws "not yet wired to transport." This feature implements the actual MCP transport layer so that `McpManager.callTool()` dispatches real requests to configured MCP servers.
+
+Additionally, this feature wires the discovery phase to use web search and WorkIQ MCP tools for gathering company and industry context, and researches how the GitHub Copilot SDK structures agent definitions and MCP tool routing to ensure architectural alignment.
+
+**Gaps addressed**: GAP-001, GAP-002, GAP-003, GAP-004, GAP-005, GAP-006 (P1), GAP-007 (P1) from `specs/003-next-spec-gaps.md`.  
+**Gaps deferred**: GAP-006 (P2, resume/checkpoint), GAP-007 (P2, `--force`), GAP-008 (P2, testRunner coverage), GAP-009 (P2, template selection) — see Out of Scope.
+
+## Clarifications
+
+### Session 2026-03-01
+
+- Q: Should P2 gaps (resume/checkpoint, --force, testRunner coverage, template selection) be included in this feature spec or deferred? → A: Explicitly defer all P2 gaps to a separate feature spec; add Out of Scope section.
+- Q: How should MCP authentication work across transport types (stdio env, HTTP SDK, WorkIQ OAuth)? → A: Defer auth design to implementation research (FR-019); determine correct pattern per transport during SDK research phase.
+- Q: Should failed MCP tool calls be retried automatically? → A: One automatic retry with backoff for transient errors (connection refused, timeout); no retry for auth/validation errors.
+- Q: What is the schema shape of DiscoveryEnrichment? → A: Flat structure with optional string arrays and nested WorkIQ insights object.
+- Q: What if the Copilot SDK doesn't provide MCP integration or agent registration patterns? → A: Research SDK first; use built-in MCP support where available; build custom transport only where SDK lacks coverage; modify existing code to align with SDK patterns.
+
+## Out of Scope
+
+The following items from `specs/003-next-spec-gaps.md` are explicitly deferred to a subsequent feature spec to limit scope risk:
+
+- **Resume/checkpoint for `sofia dev`** (P2 GAP-006) — Detect existing PoC directory and resume from last iteration instead of re-scaffolding. _Note: SDK provides native `resumeSession(sessionId)` with state stored at `~/.copilot/session-state/` — implementation complexity is lower than originally assessed (see research.md Topic 9)._
+- **`--force` flag implementation** (P2 GAP-007) — Honor the declared `--force` CLI option to delete existing output and restart.
+- **testRunner.ts coverage hardening** (P2 GAP-008) — Add spawn-based integration tests for child process spawning, timeout, and SIGTERM/SIGKILL paths.
+- **PoC template selection** (P2 GAP-009) — Define a template registry mapping plan characteristics to scaffold templates (e.g., Python/FastAPI).
+- **Generated scaffold TODOs** (P3 GAP-009) — Tracking intentional TODO markers in generated code for template quality.
+- **PTY-based E2E tests** (P3 GAP-010) — Interactive terminal tests for `sofia dev` Ctrl+C handling and spinner output.
+- **Workshop→develop phase transition** (P3 GAP-011) — Whether `workshop` should auto-invoke the Ralph loop after Plan completion.
+
+## User Scenarios & Testing _(mandatory)_
+
+### User Story 1 — MCP Tool Calls Work in Production (Priority: P1)
+
+As a facilitator running `sofia dev`, I want the Ralph Loop to create a real GitHub repository, query Context7 for library documentation, fetch Azure architecture guidance, and perform web searches when stuck — so that the PoC generation pipeline works end-to-end in a properly configured environment.
+
+**Why this priority**: Without working MCP transport, the entire PoC generation pipeline produces degraded output. Every MCP-dependent component returns hardcoded or simulated data, making the Ralph Loop significantly less effective.
+
+**Independent Test**: Configure a test environment with a mock MCP server implementing the MCP protocol, run `sofia dev` on a session with a plan referencing Azure services and npm dependencies, and verify that real tool calls are dispatched and real responses are used in the LLM prompt.
+
+**Acceptance Scenarios**:
+
+1. **Given** a configured MCP environment with GitHub MCP available, **When** the Ralph Loop scaffolds a PoC, **Then** `createRepository` dispatches a `create_repository` tool call via the MCP transport and returns the real repository URL from the server response.
+2. **Given** a configured MCP environment with Context7 available, **When** the context enricher queries library docs for a dependency (e.g., "express"), **Then** `queryContext7` dispatches `resolve-library-id` followed by `query-docs` tool calls via MCP transport and returns real documentation text.
+3. **Given** a configured MCP environment with Azure MCP available, **When** the plan references Azure services, **Then** `queryAzureMcp` dispatches a `documentation` tool call and returns real architecture guidance.
+4. **Given** all MCP servers are unavailable, **When** the Ralph Loop runs, **Then** all adapters degrade gracefully to local fallbacks (local scaffold, static context strings) without errors or crashes.
+
+---
+
+### User Story 2 — GitHub MCP Pushes Real File Content (Priority: P1)
+
+As a facilitator, I want the Ralph Loop to push actual file content (not empty strings) to the GitHub repository after each iteration, so that the remote repository always reflects the current state of the PoC.
+
+**Why this priority**: Even with MCP transport working, pushing empty files makes the GitHub integration useless. This is a data-flow bug that must be fixed alongside the transport layer.
+
+**Independent Test**: Run a Ralph Loop iteration that modifies files, verify that `pushFiles` sends the actual file content read from disk to the MCP server.
+
+**Acceptance Scenarios**:
+
+1. **Given** the Ralph Loop has applied code changes in an iteration, **When** `pushFiles` is called, **Then** each file in the push request contains the actual content read from disk (not empty strings).
+2. **Given** a file path is outside the output directory, **When** `pushFiles` prepares the file list, **Then** that file is skipped with a warning logged.
+
+---
+
+### User Story 3 — Discovery Phase Uses Web Search and WorkIQ (Priority: P2)
+
+As a facilitator running a discovery workshop, I want sofIA to optionally search the web for recent company news, competitor activity, and industry trends after I provide company information, so that the ideation phase is informed by current market context.
+
+**Why this priority**: Enriching the discovery phase with real-world context improves PoC relevance, but the core pipeline works without it. This is an enhancement to workshop quality.
+
+**Independent Test**: Start a workshop session, provide company and team information, verify that sofIA offers to search for relevant context and stores the results in the session for later phases.
+
+**Acceptance Scenarios**:
+
+1. **Given** the user has described their business in Step 1, **When** the discovery agent processes this input, **Then** it offers to search the web for recent news about the company, competitors, and industry trends.
+2. **Given** the user consents to web search enrichment, **When** the search completes, **Then** the results are stored in the session's discovery state and are available to inform ideation and planning phases.
+3. **Given** web search MCP is unavailable, **When** the discovery agent tries to enrich context, **Then** it skips enrichment gracefully with a message explaining that web search is not available.
+
+---
+
+### User Story 4 — WorkIQ Integration for Internal Context (Priority: P3)
+
+As a facilitator, I want sofIA to optionally query WorkIQ to analyze internal documentation, meeting patterns, and team expertise, so that the PoC is aligned with the team's actual strengths and collaboration patterns.
+
+**Why this priority**: WorkIQ provides valuable internal context but requires Microsoft 365 access and admin consent. It's an optional enhancement that adds significant value when available but is not required for core functionality.
+
+**Independent Test**: With WorkIQ configured and authorized, verify that sofIA asks permission before querying, returns meaningful team insights, and stores them in the session.
+
+**Acceptance Scenarios**:
+
+1. **Given** WorkIQ is configured and available, **When** the discovery phase gathers team information, **Then** sofIA asks the user for explicit permission before querying WorkIQ.
+2. **Given** the user grants WorkIQ permission, **When** WorkIQ returns team insights, **Then** these insights are stored in the session and surfaced during ideation to help shape the PoC approach.
+3. **Given** the user declines WorkIQ access or WorkIQ is unavailable, **When** the discovery phase continues, **Then** it proceeds normally without internal context, with no errors.
+
+---
+
+### User Story 5 — Copilot SDK Agent Architecture Alignment (Priority: P2)
+
+As a developer, I want the MCP transport layer to align with how the GitHub Copilot SDK structures agent definitions and tool routing, so that sofIA's agents can be registered and orchestrated through the SDK's native patterns rather than custom dispatch.
+
+**Why this priority**: Aligning with SDK patterns now avoids a costly refactor later and enables sofIA to leverage SDK features like built-in authentication, tool approval flows, and session management.
+
+**Independent Test**: Verify that the agent definitions follow Copilot SDK conventions and that MCP tool calls flow through the SDK's expected tool-calling interface.
+
+**Acceptance Scenarios**:
+
+1. **Given** the Copilot SDK provides native `mcpServers` support in `SessionConfig`, **When** `createSession()` is called with MCP server configurations from `.vscode/mcp.json`, **Then** the SDK manages server lifecycle (spawn/connect, JSON-RPC, tool dispatch) for LLM-initiated tool calls without custom transport code.
+2. **Given** adapters (GitHub, Context7, Azure) need deterministic programmatic tool calls, **When** `McpManager.callTool()` is invoked, **Then** it routes through the custom transport layer (`StdioMcpTransport` / `HttpMcpTransport`) since the SDK's `executeToolCall` is private and cannot be called directly from application code.
+3. **Given** the SDK defines agent registration patterns, **When** sofIA's discovery/ideation/design/select/plan agents are initialized, **Then** they follow SDK conventions for capability declaration and tool access.
+
+### Dual-Lifecycle Limitation
+
+**⚠️ Known limitation**: The Copilot SDK's `mcpServers` support and the custom `McpTransport` layer manage MCP server connections independently. If the same server (e.g., `context7`) is used both by the LLM during conversation turns (SDK-managed) and by a programmatic adapter call (custom transport-managed), **two separate subprocess instances** will be spawned. This is acceptable for Feature 003 scope because:
+
+- The two paths serve fundamentally different call patterns (LLM-driven vs. deterministic).
+- MCP subprocesses are lightweight (Node.js CLI tools via npx).
+- HTTP servers (GitHub, Microsoft Docs) are stateless and unaffected.
+
+This limitation should be revisited in a future feature if subprocess resource usage becomes a concern, potentially by sharing a single subprocess instance between the SDK and custom transport layers.
+
+---
+
+### Edge Cases
+
+- What happens when an MCP server disconnects mid-tool-call? The transport MUST timeout and return a classified error that adapters handle gracefully.
+- How does the system handle MCP servers returning malformed JSON responses? The transport MUST parse defensively and throw typed errors classifiable by `classifyMcpError()`.
+- What happens when Context7 `resolve-library-id` succeeds but `query-docs` fails? Return whatever partial context was gathered.
+- What happens when the GitHub MCP `create_repository` call creates the repo but the subsequent `push_files` call fails? The repository URL MUST still be recorded; the push failure is a recoverable error for the next iteration.
+- What if WorkIQ requires re-authentication mid-session? Surface a clear message to the user and skip enrichment for that query.
+
+## Requirements _(mandatory)_
+
+### Functional Requirements
+
+#### MCP Transport Layer (GAP-001)
+
+- **FR-001**: `McpManager` MUST implement a working `callTool(serverName, toolName, args)` method that dispatches real tool calls to configured MCP servers and returns structured results.
+- **FR-002**: The transport layer MUST support both `stdio` (subprocess-based) and `http` (URL-based) MCP server configurations as defined in `.vscode/mcp.json`.
+- **FR-003**: Tool calls MUST include configurable timeouts (default: 30 seconds for data queries, 60 seconds for repository operations).
+- **FR-004**: The transport MUST handle connection failures, timeouts, and malformed responses by throwing typed errors that callers can classify using `classifyMcpError()`.
+- **FR-004a**: The transport MUST automatically retry once with exponential backoff for transient errors (connection refused, timeout, server unavailable). Auth failures and validation errors MUST NOT be retried.
+- **FR-005**: The transport MUST support the MCP protocol's request/response format, including JSON-RPC message framing for stdio servers.
+
+#### GitHub MCP Integration (GAP-002, GAP-003)
+
+- **FR-006**: `GitHubMcpAdapter.createRepository()` MUST dispatch a `create_repository` tool call via the MCP transport and extract the repository URL from the real response.
+- **FR-007**: `GitHubMcpAdapter.pushFiles()` MUST dispatch a `push_files` tool call via the MCP transport, sending actual file content read from disk.
+- **FR-008**: The Ralph Loop MUST read file content from disk before passing it to `pushFiles()`, replacing the current empty-string behavior (GAP-003).
+- **FR-009**: Both GitHub adapter methods MUST degrade gracefully when the MCP transport is unavailable, returning `{ available: false, reason }`.
+
+#### Context Enrichment MCP Integration (GAP-004)
+
+- **FR-010**: `McpContextEnricher.queryContext7()` MUST dispatch `resolve-library-id` and `query-docs` tool calls to the Context7 MCP server and return real documentation text.
+- **FR-011**: `McpContextEnricher.queryAzureMcp()` MUST dispatch a `documentation` tool call to the Azure MCP server with architecture keywords and return real guidance.
+- **FR-012**: `McpContextEnricher.queryWebSearch()` MUST dispatch a search tool call when the Ralph Loop is stuck (2+ consecutive iterations with same failures).
+- **FR-013**: All enrichment methods MUST fall back to static/empty context when their respective MCP servers are unavailable, with no impact on the Ralph Loop's ability to continue iterating.
+
+#### Discovery Phase Enrichment (GAP-005)
+
+- **FR-014**: The discovery phase SHOULD offer web search enrichment after the user provides company and team information in Step 1.
+- **FR-015**: Web search results (company news, competitor activity, industry trends) MUST be stored in the session state for use in subsequent phases.
+- **FR-016**: WorkIQ integration MUST request explicit user permission before accessing Microsoft 365 data. _Note: SDK provides `onPermissionRequest` handler as an alternative — evaluated and deferred in favor of `io.prompt()` for consistency with other interactive prompts (see research.md Topic 8)._
+- **FR-017**: WorkIQ-derived insights (team collaboration patterns, documentation gaps, expertise areas) MUST be stored in the session state when the user consents.
+- **FR-018**: Both web search and WorkIQ enrichment MUST be optional — the discovery phase MUST function normally without them.
+
+#### Copilot SDK Alignment (GAP-006, GAP-007)
+
+- **FR-019**: Before implementing custom MCP transport, the team MUST research the GitHub Copilot SDK's built-in MCP support and tool-calling capabilities. Where the SDK provides native MCP integration (tool dispatch, authentication, protocol handling), the implementation MUST use the SDK's mechanisms rather than building custom transport. Custom stdio/HTTP transport MUST only be built for capabilities the SDK does not cover. This research MUST also determine the authentication model for each transport type. Findings MUST be documented in a research note under `specs/003-mcp-transport-integration/research.md`.
+- **FR-020**: Agent definitions (discovery, ideation, design, select, plan, develop) MUST be verified to follow Copilot SDK conventions for agent registration and capability declaration. Where SDK patterns (e.g., `customAgents`, `skillDirectories`) offer advantages over the current implementation, they SHOULD be evaluated and adopted if beneficial. Research findings (research.md Topic 7) confirmed no structural refactoring is currently required; SDK `customAgents` and `skillDirectories` evaluated and deferred (research.md Topic 9).
+
+#### SDK Hooks & Transparency (Constitution Principle VIII)
+
+- **FR-021**: The system MUST use the Copilot SDK's `onPreToolUse` and `onPostToolUse` hooks to emit tool-call activity events (tool name, start/end, duration) visible via the CLI spinner or activity log, satisfying Constitution Principle VIII ("Users MUST always see the current execution state"). Findings documented in research.md Topic 8.
+- **FR-022**: The system SHOULD use the Copilot SDK's `onErrorOccurred` hook to centralize error handling for LLM-conversation-path MCP failures, complementing `classifyMcpError()` which handles the custom transport path. FR-004's error classification covers programmatic adapter calls only; SDK-managed tool call errors require the `onErrorOccurred` hook.
+- **FR-023**: Ralph Loop sessions SHOULD wire the SDK's `infiniteSessions` config (with `backgroundCompactionThreshold` and `bufferExhaustionThreshold`) to prevent context window exhaustion during extended multi-iteration conversations. Without this, long Ralph Loop runs risk silently truncating important context (see research.md Topic 9).
+- **FR-024**: The system SHOULD subscribe to SDK events (e.g., `assistant.usage` for token tracking) to provide real-time progress and usage transparency in the CLI, per Constitution Principle VIII. Token counts logged at `debug` level and optionally surfaced in the CLI spinner.
+
+### Key Entities
+
+- **McpTransport**: Abstraction over the communication channel to an MCP server. Handles JSON-RPC framing for stdio servers and HTTP requests for HTTP servers. Manages connection lifecycle (connect, call, disconnect).
+- **ToolCallRequest**: Structured request containing server name, tool name, and arguments. Includes timeout and retry configuration.
+- **ToolCallResponse**: Structured response from an MCP server containing the tool result as parsed JSON, or an error with classification.
+- **DiscoveryEnrichment**: Optional context gathered during the discovery phase, stored in the session for downstream phases. Schema:
+  ```typescript
+  {
+    webSearchResults?: string;        // raw search summary text
+    companyNews?: string[];            // recent news headlines/snippets
+    competitorInfo?: string[];         // competitor activity summaries
+    industryTrends?: string[];         // industry trend descriptions
+    workiqInsights?: {
+      teamExpertise?: string[];        // identified team skill areas
+      collaborationPatterns?: string[];// meeting/communication patterns
+      documentationGaps?: string[];   // areas lacking documentation
+    };
+  }
+  ```
+
+## Success Criteria _(mandatory)_
+
+### Measurable Outcomes
+
+- **SC-003-001**: All four MCP-dependent components (GitHub adapter, Context7 enricher, Azure enricher, web search enricher) successfully dispatch real tool calls and process real responses in a configured environment.
+- **SC-003-002**: When any MCP server is unavailable, the system operates with the same graceful degradation behavior as the current stub implementation — no crashes, clear fallback messages, functional Ralph Loop output.
+- **SC-003-003**: GitHub `pushFiles` sends actual file content to the MCP server, verified by inspecting the tool call arguments in integration tests.
+- **SC-003-004**: The Ralph Loop completes at least one end-to-end run where Context7 documentation and web search results measurably improve the LLM's ability to fix failing tests (measured by comparing iteration counts with and without enrichment).
+- **SC-003-005**: Discovery phase web search enrichment retrieves relevant context for at least 3 out of 5 test company descriptions, measured by keyword relevance in search results.
+- **SC-003-006**: WorkIQ integration, when authorized, returns team insights within 10 seconds and stores them in the session without errors.
+- **SC-003-007**: All MCP tool calls complete within their configured timeouts (30s for queries, 60s for repository operations) or return a classified timeout error.
+
+## Assumptions
+
+- The GitHub Copilot SDK (`@github/copilot-sdk` v0.1.28+) is the primary implementation path for MCP transport. FR-019 research MUST determine what the SDK provides natively before building any custom transport. Custom stdio/HTTP transport is only built for gaps in SDK coverage.
+- MCP servers configured in `.vscode/mcp.json` follow the standard MCP protocol (JSON-RPC 2.0 over stdio, or HTTP endpoints).
+- The authentication model for each MCP transport type is not yet determined. FR-019 research will establish whether stdio servers inherit environment credentials, whether HTTP servers use Copilot SDK auth, and whether WorkIQ requires its own OAuth flow. No custom auth abstraction should be built until this research is complete.
+- Context7 and Azure MCP servers are publicly accessible npm packages (`@upstash/context7-mcp`, `@azure/mcp`) that can be spawned as subprocesses — unless the SDK provides a different mechanism for spawning/connecting.
+- WorkIQ requires Microsoft 365 tenant access with admin consent, as documented in the WorkIQ Admin Instructions.
+- The `web.search` capability is either built into the Copilot SDK or available as an MCP tool — research is needed to confirm the exact integration path.
+- Live integration tests for MCP servers will be gated behind environment variables (e.g., `SOFIA_LIVE_MCP_TESTS=true`) to avoid CI failures when servers are not available.
+
+## Dependencies
+
+- **Feature 001**: Session model, workshop phases, plan outputs
+- **Feature 002**: Ralph Loop, GitHub adapter, context enricher, PoC scaffolder
+- **External**: `@github/copilot-sdk` v0.1.28+, MCP servers configured in `.vscode/mcp.json`