sofia-cli 0.1.1

package/specs/003-mcp-transport-integration/plan.md

@@ -0,0 +1,114 @@
+# Implementation Plan: MCP Transport Integration
+
+**Branch**: `003-mcp-transport-integration` | **Date**: 2026-03-01 | **Spec**: [specs/003-mcp-transport-integration/spec.md](spec.md)
+**Input**: Feature specification from `/specs/003-mcp-transport-integration/spec.md`
+
+**Note**: This plan was generated by the `/speckit.plan` command.
+
+## Summary
+
+Implement the real MCP transport layer that connects `McpManager.callTool()` to actual MCP server processes and HTTP endpoints, replacing all stubs and placeholder implementations introduced in Feature 002.
+
+The four currently-stubbed MCP integrations — GitHub adapter (repo creation + file push), Context7 enricher (library docs), Azure MCP enricher (cloud guidance), and web search enricher (stuck-iteration research) — will be wired to real transport. Research (FR-019) confirmed that `@github/copilot-sdk` v0.1.28 provides **native MCP server management** via `SessionConfig.mcpServers` for LLM-initiated tool calls during conversation turns, but the SDK's `executeToolCall()` is private — so a custom `mcpTransport.ts` (stdio + HTTP) is built for programmatic adapter calls (GitHub, Context7, Azure) that bypass the LLM.
+
+Additionally: the discovery phase gains optional web search and WorkIQ enrichment; `pushFiles` is fixed to include an initial post-scaffold push; and agent definitions are confirmed aligned with SDK patterns (no refactoring required).
+
+## Technical Context
+
+**Language/Version**: Node.js (>=20 LTS) with TypeScript 5.3 (ES2022 target, ESM)
+**Primary Dependencies**: `@github/copilot-sdk` ^0.1.28 (primary MCP + LLM surface), `zod` ^4 (response validation), `pino` ^8 (structured logging), `@upstash/context7-mcp` (stdio MCP subprocess), `@azure/mcp` (stdio MCP subprocess), `@microsoft/workiq` (stdio MCP subprocess)
+**Storage**: Filesystem — session JSON at `.sofia/sessions/<id>.json`, PoC output at `./poc/<sessionId>/`; MCP config from `.vscode/mcp.json`
+**Testing**: Vitest ^4 with `@vitest/coverage-v8`; unit, integration, e2e directories; live MCP tests gated behind `SOFIA_LIVE_MCP_TESTS=true`
+**Target Platform**: Linux/macOS CLI (Windows via WSL); MCP subprocesses spawned via `npx`
+**Project Type**: CLI application — extending existing `sofiacli` binary; MCP transport is an internal service module, not a public API
+**Performance Goals**: Tool calls complete within configured timeout (30s queries, 60s repo operations); one automatic retry with exponential backoff for transient errors; MCP subprocess startup <5s
+**Constraints**: No secrets/tokens in logs; SDK-native transport used where available; custom transport only for SDK gaps; live tests gated by env var; auth model per transport determined by FR-019 research
+**Scale/Scope**: Single-user CLI; up to 4 concurrent MCP server connections; typical session makes <50 tool calls
+
+## Constitution Check
+
+_GATE: Must pass before Phase 0 research. Re-check after Phase 1 design._
+
+- **Outcome-first discovery**: ✅ Working MCP transport is the critical path to end-to-end PoC generation — the measurable outcome is a real GitHub repo with real code, created and enriched by live MCP tool calls.
+- **Secure-by-default**: ✅ No tokens or secrets logged; WorkIQ requires explicit user consent before accessing M365 data; MCP subprocess env vars inherit from shell (not hardcoded); least privilege per server config.
+- **Node.js + TypeScript**: ✅ All new code extends the existing TypeScript ESM codebase; Copilot SDK used as primary integration surface.
+- **MCP-first**: ✅ This entire feature IS the MCP integration — every external call routes through MCP protocol rather than ad-hoc HTTP. WorkIQ uses its MCP interface. Web search uses the `web.search` MCP tool or Azure AI Foundry bridge already in `src/mcp/webSearch.ts`.
+- **Test-first (NON-NEGOTIABLE)**: ✅ Transport layer, retry logic, error classification, all adapter methods, and discovery enrichment flows require Red → Green → Review. Live integration tests gated behind `SOFIA_LIVE_MCP_TESTS=true`. Mock MCP servers used in unit/integration tests.
+- **CLI transparency**: ✅ SDK `onPreToolUse`/`onPostToolUse` hooks emit tool-call activity to the CLI spinner; `assistant.usage` events tracked for token transparency; retry attempts logged at warn level; graceful degradation messages include what was skipped and why.
+
+## Project Structure
+
+### Documentation (this feature)
+
+```text
+specs/003-mcp-transport-integration/
+├── plan.md              # This file
+├── research.md          # Phase 0 output — Copilot SDK MCP patterns, transport options, auth model
+├── data-model.md        # Phase 1 output — McpTransport, ToolCallRequest, DiscoveryEnrichment entities
+├── quickstart.md        # Phase 1 output — developer setup and first tool call walkthrough
+├── contracts/           # Phase 1 output
+│   ├── mcp-transport.md     # McpManager.callTool() contract and transport abstraction
+│   ├── github-adapter.md    # GitHubMcpAdapter real integration contract
+│   ├── context-enricher.md  # McpContextEnricher real integration contract
+│   └── discovery-enricher.md # DiscoveryEnrichment session storage contract
+└── tasks.md             # Phase 2 output (created by /speckit.tasks — NOT by /speckit.plan)
+```
+
+### Source Code (repository root)
+
+```text
+src/
+├── mcp/
+│   ├── mcpManager.ts          # MODIFIED: implement callTool() with real transport dispatch
+│   ├── mcpTransport.ts        # NEW: stdio and HTTP transport implementations
+│   ├── retryPolicy.ts         # NEW: exponential backoff retry logic (transient errors only)
+│   └── webSearch.ts           # EXISTING: Azure AI Foundry web search tool (no change)
+├── develop/
+│   ├── githubMcpAdapter.ts    # MODIFIED: createRepository() and pushFiles() use real callTool()
+│   ├── mcpContextEnricher.ts  # MODIFIED: queryContext7(), queryAzureMcp(), queryWebSearch() real
+│   ├── ralphLoop.ts           # MODIFIED: fix pushFiles to read actual file content from disk
+│   ├── codeGenerator.ts       # EXISTING: no changes needed
+│   ├── pocScaffolder.ts       # EXISTING: no changes needed
+│   └── testRunner.ts          # EXISTING: no changes needed
+├── phases/
+│   ├── phaseHandlers.ts       # MODIFIED: wire discovery enrichment after Step 1 collection
+│   └── discoveryEnricher.ts   # NEW: web search + WorkIQ enrichment for discovery phase
+├── shared/
+│   └── schemas/
+│       └── session.ts         # MODIFIED: add DiscoveryEnrichment to DiscoveryState schema
+
+tests/
+├── unit/
+│   ├── mcp/
+│   │   ├── mcpManager.spec.ts       # MODIFIED: add callTool() unit tests
+│   │   ├── mcpTransport.spec.ts     # NEW: stdio + HTTP transport unit tests (mock subprocess)
+│   │   └── retryPolicy.spec.ts      # NEW: retry logic unit tests
+│   └── develop/
+│       ├── githubMcpAdapter.spec.ts  # MODIFIED: add real integration tests with mock transport
+│       ├── mcpContextEnricher.spec.ts # MODIFIED: add real integration tests with mock transport
+│       └── ralphLoop.spec.ts         # MODIFIED: fix pushFiles content test
+│   └── phases/
+│       └── discoveryEnricher.spec.ts # NEW: discovery enrichment unit tests
+├── integration/
+│   ├── mcpTransportFlow.spec.ts     # NEW: end-to-end transport with mock MCP server
+│   └── discoveryEnrichmentFlow.spec.ts # NEW: discovery + enrichment integration
+└── e2e/
+    └── mcpLive.spec.ts              # NEW: live MCP smoke tests (SOFIA_LIVE_MCP_TESTS=true gate)
+```
+
+**Structure Decision**: Single project, extending the existing `src/` layout. New transport code lives in `src/mcp/` alongside the existing `mcpManager.ts` and `webSearch.ts`. Discovery enrichment is a new module in `src/phases/`. All other changes are modifications to Feature 002 files.
+
+## Post-Design Constitution Re-Check
+
+_All 6 gates re-evaluated after Phase 1 design. No violations._
+
+- **Outcome-first discovery**: ✅ Contracts tie every transport call to a concrete user-visible outcome (real repo URL, real docs, real search results).
+- **Secure-by-default**: ✅ WorkIQ permission gate enforced in `discoveryEnricher.ts`; no env-var secrets hardcoded; subprocess stdio does not log raw credentials.
+- **Node.js + TypeScript**: ✅ All new modules follow existing ESM + strict TS patterns; Zod schemas for all response validation.
+- **MCP-first**: ✅ `mcpTransport.ts` IS the MCP protocol implementation; no ad-hoc HTTP side channels.
+- **Test-first (NON-NEGOTIABLE)**: ✅ Every new module has a spec file defined in the structure above. Mock MCP servers used in unit/integration; live tests gated.
+- **CLI transparency**: ✅ SDK `onPreToolUse`/`onPostToolUse` hooks emit tool-call activity visible via spinner; `assistant.usage` events logged for token transparency; retry attempts log warn-level messages with backoff delay; degradation messages specify which MCP server was skipped.
+
+## Complexity Tracking
+
+No constitution violations — all gates pass (both pre-research and post-design checks).

package/specs/003-mcp-transport-integration/quickstart.md

@@ -0,0 +1,311 @@
+# Developer Quickstart: MCP Transport Integration
+
+**Feature**: 003-mcp-transport-integration  
+**Date**: 2026-03-01  
+**Audience**: Engineers implementing or reviewing this feature
+
+---
+
+## Prerequisites
+
+- Node.js >= 20 LTS
+- `npm install` run at repo root
+- Feature 002 (`002-poc-generation`) merged to `main`
+- Working branch: `003-mcp-transport-integration`
+- (Optional) `GITHUB_TOKEN` env var set to a PAT with `repo` scope for live GitHub MCP tests
+- (Optional) Azure subscription credentials for live Azure MCP tests
+
+---
+
+## 1. Clone and Set Up
+
+```bash
+git clone <repo>
+cd sofIA-cli
+git checkout 003-mcp-transport-integration
+npm install
+```
+
+Verify the test suite passes before making any changes (Feature 002 baseline):
+
+```bash
+npm run test
+```
+
+Expected: all unit and integration tests pass. e2e tests may require live services.
+
+---
+
+## 2. Understand the Transport Layer
+
+The MCP transport lives in `src/mcp/`:
+
+```
+src/mcp/
+├── mcpManager.ts      # Config loader + McpManager class (callTool to be implemented)
+├── mcpTransport.ts    # NEW: StdioMcpTransport, HttpMcpTransport, createTransport()
+├── retryPolicy.ts     # NEW: withRetry() helper for transient error handling
+└── webSearch.ts       # Azure AI Foundry bridge (existing, no changes)
+```
+
+### How a tool call flows
+
+```
+adapter.createRepository()
+  └── mcpManager.callTool('github', 'create_repository', args)
+        └── withRetry(...)
+              └── HttpMcpTransport.callTool('create_repository', args, 60_000)
+                    └── fetch('https://api.githubcopilot.com/mcp/', { method: 'POST', ... })
+                          └── returns ToolCallResponse
+```
+
+For stdio servers (Context7, Azure, WorkIQ):
+
+```
+enricher.queryContext7(['express', 'zod'])
+  └── mcpManager.callTool('context7', 'resolve-library-id', { libraryName: 'express' })
+        └── withRetry(...)
+              └── StdioMcpTransport.callTool('resolve-library-id', args, 30_000)
+                    └── writes JSON-RPC to subprocess stdin
+                    └── reads JSON-RPC from subprocess stdout
+```
+
+---
+
+## 3. First Tool Call (Development Walkthrough)
+
+### Step A: Write the failing test (RED)
+
+Before writing any implementation, add a failing test:
+
+```typescript
+// tests/unit/mcp/mcpTransport.spec.ts
+
+import { describe, it, expect } from 'vitest';
+import { HttpMcpTransport } from '../../../src/mcp/mcpTransport.js';
+
+describe('HttpMcpTransport', () => {
+  it('calls the server URL with JSON-RPC method tools/call', async () => {
+    const calls: RequestInit[] = [];
+    const mockFetch = async (url: string, init: RequestInit) => {
+      calls.push(init);
+      return new Response(JSON.stringify({
+        jsonrpc: '2.0',
+        id: 1,
+        result: { content: [{ type: 'text', text: '{"html_url":"https://github.com/org/repo"}' }] }
+      }), { status: 200 });
+    };
+
+    const transport = new HttpMcpTransport(
+      { name: 'github', type: 'http', url: 'https://api.githubcopilot.com/mcp/' },
+      mockFetch as typeof fetch,
+    );
+
+    const result = await transport.callTool('create_repository', { name: 'test' }, 30_000);
+
+    expect(calls).toHaveLength(1);
+    const body = JSON.parse(calls[0].body as string);
+    expect(body.method).toBe('tools/call');
+    expect(body.params.name).toBe('create_repository');
+    expect(result.content).toContain('html_url');
+  });
+});
+```
+
+Run: `npm run test:unit` → **should fail** (module doesn't exist yet).
+
+### Step B: Implement (GREEN)
+
+Create `src/mcp/mcpTransport.ts` with `HttpMcpTransport` satisfying the test.
+
+Run: `npm run test:unit` → **should pass**.
+
+### Step C: Review
+
+Checklist:
+- [ ] Are error paths tested (401, timeout, malformed JSON)?
+- [ ] Is the retry policy exercised (separate `retryPolicy.spec.ts`)?
+- [ ] Does `McpManager.callTool()` integration test pass?
+
+---
+
+## 4. MCP Configuration
+
+The transport reads server configurations from `.vscode/mcp.json` (already present in the repo):
+
+```json
+{
+  "servers": {
+    "github": { "type": "http", "url": "https://api.githubcopilot.com/mcp/" },
+    "context7": { "command": "npx", "args": ["-y", "@upstash/context7-mcp"] },
+    "azure": { "command": "npx", "args": ["-y", "@azure/mcp", "server", "start"] },
+    "workiq": { "command": "npx", "args": ["-y", "@microsoft/workiq", "mcp"] }
+  }
+}
+```
+
+`McpManager` loads this file via `loadMcpConfig()`. In tests, pass a custom `McpConfig` directly:
+
+```typescript
+const config: McpConfig = {
+  servers: {
+    github: { name: 'github', type: 'http', url: 'https://api.githubcopilot.com/mcp/' },
+  },
+};
+const manager = new McpManager(config);
+manager.markConnected('github');
+```
+
+---
+
+## 5. Running Tests
+
+### Unit tests only (fast, no live services)
+
+```bash
+npm run test:unit
+```
+
+### Integration tests (mock MCP servers, no live services)
+
+```bash
+npm run test:integration
+```
+
+### Live MCP smoke tests (requires credentials)
+
+```bash
+SOFIA_LIVE_MCP_TESTS=true npm run test:live
+```
+
+> **Note**: Live tests will spawn real MCP subprocesses (`npx @upstash/context7-mcp`, etc.) and make real HTTP calls. Ensure `GITHUB_TOKEN` is set for GitHub MCP tests.
+
+### Full test suite
+
+```bash
+npm test
+```
+
+---
+
+## 6. Environment Variables
+
+| Variable | Required | Description |
+|----------|----------|-------------|
+| `GITHUB_TOKEN` | For live GitHub MCP | Personal access token with `repo` scope |
+| `AZURE_SUBSCRIPTION_ID` | For live Azure MCP | Azure subscription ID |
+| `AZURE_TENANT_ID` | For live Azure MCP | Azure tenant ID |
+| `AZURE_AI_FOUNDRY_ENDPOINT` | For web search | Azure AI Foundry endpoint URL |
+| `AZURE_AI_FOUNDRY_API_KEY` | For web search | Azure AI Foundry API key |
+| `SOFIA_LIVE_MCP_TESTS` | To enable live tests | Set to `true` to run live smoke tests |
+
+---
+
+## 7. Key Files to Modify
+
+### New files
+
+| File | Purpose |
+|------|---------|
+| `src/mcp/mcpTransport.ts` | `McpTransport` interface + `StdioMcpTransport` + `HttpMcpTransport` + `createTransport()` |
+| `src/mcp/retryPolicy.ts` | `withRetry()` helper |
+| `src/phases/discoveryEnricher.ts` | `DiscoveryEnricher` class |
+| `tests/unit/mcp/mcpTransport.spec.ts` | Transport unit tests |
+| `tests/unit/mcp/retryPolicy.spec.ts` | Retry policy unit tests |
+| `tests/unit/phases/discoveryEnricher.spec.ts` | Discovery enricher unit tests |
+| `tests/integration/mcpTransportFlow.spec.ts` | End-to-end transport integration test |
+| `tests/e2e/mcpLive.spec.ts` | Live smoke tests (gated by `SOFIA_LIVE_MCP_TESTS`) |
+
+### Modified files
+
+| File | Change |
+|------|--------|
+| `src/mcp/mcpManager.ts` | Add `callTool()` implementation + transport registry |
+| `src/develop/githubMcpAdapter.ts` | Remove stub; use real `callTool()` |
+| `src/develop/mcpContextEnricher.ts` | Remove stub; use real `callTool()` |
+| `src/develop/ralphLoop.ts` | Add post-scaffold push; fix file content flow |
+| `src/phases/phaseHandlers.ts` | Wire discovery enrichment after Step 1 |
+| `src/shared/schemas/session.ts` | Add `DiscoveryEnrichment` + extend `DiscoveryState` |
+
+---
+
+## 8. Common Patterns
+
+### Making a tool call in an adapter
+
+```typescript
+// In any adapter or enricher:
+try {
+  const response = await this.mcpManager.callTool('context7', 'resolve-library-id', {
+    libraryName: 'express',
+  });
+  const libraryId = response.libraryId as string;
+  // ... use libraryId
+} catch (err) {
+  const errorClass = classifyMcpError(err);
+  logger.warn({ errorClass, err }, 'Context7 resolve-library-id failed');
+  // return fallback value
+}
+```
+
+### Writing a test with a fake transport
+
+```typescript
+import { McpManager, loadMcpConfig } from '../../../src/mcp/mcpManager.js';
+
+const manager = new McpManager({ servers: {
+  context7: { name: 'context7', type: 'stdio', command: 'npx', args: [] }
+}});
+
+// Inject fake transport (via a test seam or vi.mock)
+manager.markConnected('context7');
+vi.spyOn(manager, 'callTool').mockResolvedValueOnce({ libraryId: '/expressjs/express' });
+```
+
+### Gating a live test
+
+```typescript
+import { describe, it } from 'vitest';
+
+const LIVE = process.env.SOFIA_LIVE_MCP_TESTS === 'true';
+
+describe.skipIf(!LIVE)('Live: GitHub MCP smoke test', () => {
+  it('creates and deletes a test repository', async () => {
+    // ... real MCP call
+  });
+});
+```
+
+---
+
+## 9. Troubleshooting
+
+### "MCP server 'github' is not available"
+
+The server is in `mcp.json` config but not yet connected. Call `manager.markConnected('github')` in your test setup, or verify the real transport initialized successfully.
+
+### Stdio subprocess hangs on startup
+
+The `initialize` handshake times out after 5 seconds. Check that the subprocess command is correct in `mcp.json` and that `npx` can download the package (requires internet access in live test environment).
+
+### Live test fails with 401 on GitHub MCP
+
+Set `GITHUB_TOKEN` to a valid PAT. Tokens must have `repo` scope. Check token expiry.
+
+### Context7 returns empty docs
+
+The `resolve-library-id` tool may return a libraryId, but `query-docs` returns empty content for some packages. This is expected behavior — the enricher falls back to the npm link.
+
+---
+
+## 10. Reference
+
+- [MCP Transport Contract](./contracts/mcp-transport.md)
+- [GitHub Adapter Contract](./contracts/github-adapter.md)
+- [Context Enricher Contract](./contracts/context-enricher.md)
+- [Discovery Enricher Contract](./contracts/discovery-enricher.md)
+- [Data Model](./data-model.md)
+- [Research Notes](./research.md)
+- [Feature 002 Plan](../002-poc-generation/plan.md) — upstream feature context
+- [MCP Protocol Specification](https://modelcontextprotocol.io/specification) — JSON-RPC 2.0 framing
+- [`.vscode/mcp.json`](../../.vscode/mcp.json) — server configurations