sofia-cli 0.1.1

package/specs/006-workshop-extraction-fixes/spec.md

@@ -0,0 +1,210 @@
+# Feature Specification: Workshop Phase Extraction & Tool Wiring Fixes
+
+**Feature Branch**: `006-workshop-extraction-fixes`  
+**Created**: 2026-03-04  
+**Status**: Draft  
+**Upstream Dependency**: specs/001-cli-workshop-rebuild/spec.md, specs/003-mcp-transport-integration/spec.md, specs/005-ai-search-deploy/spec.md  
+**Input**: User description: "Fix workshop phase extraction failures, lazy web search config, MCP tool wiring, context window management, and export completeness identified in the Zava Industries full-session assessment"
+
+## Overview
+
+A full end-to-end workshop session (the Zava Industries assessment — 6 phases, 48 turns, ~13 minutes) revealed five systemic bugs that prevent sofIA from extracting structured data from LLM responses and from using MCP tools during the workshop flow. The assessment scored **53%** (59/126 testable checks passed). The core conversational quality is excellent (rated 4–5/5 across phases), but the pipeline between LLM output and structured session state is broken for all phases except Discover's `businessContext`.
+
+This feature addresses all five bugs discovered in that assessment, plus related gaps in export completeness and context management.
+
+### Bugs Addressed
+
+| Bug ID  | Severity | Summary                                                                                                                                                                                                                                                                                                                                                          |
+| ------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| BUG-001 | High     | `isWebSearchConfigured()` checks `process.env` at function call time, but `.env` may not be loaded yet if the module was imported before `envLoader.ts` ran — making web search appear unconfigured                                                                                                                                                              |
+| BUG-002 | High     | Phase extractors (`extractIdeas`, `extractEvaluation`, `extractSelection`, `extractPlan`, `extractPocState`) only find data inside JSON code blocks; the LLM often produces structured information in prose/tables/markdown without a JSON block, leaving `session.ideas`, `session.evaluation`, `session.selection`, `session.plan`, and `session.poc` all null |
+| BUG-003 | Medium   | Select phase hit a 120-second SDK `sendAndWait()` timeout when prior phases accumulated ~38 turns of context — likely a context window or processing overload issue                                                                                                                                                                                              |
+| BUG-004 | Medium   | Export only produces `discover.md` because the export writer requires populated structured fields (e.g., `session.ideas`) to generate phase files, and those fields are empty due to BUG-002                                                                                                                                                                     |
+| BUG-005 | Medium   | MCP tools (web search, WorkIQ, Context7, Azure MCP) are not wired into the workshop flow — `workshopCommand.ts` does not create an `McpManager` or pass `webSearchClient`/MCP config to phase handlers beyond Discover                                                                                                                                           |
+
+## User Scenarios & Testing _(mandatory)_
+
+### User Story 1 — Phase data is reliably extracted from LLM responses (Priority: P1)
+
+As a facilitator completing a multi-phase workshop, I want sofIA to reliably capture structured artifacts (ideas, evaluation, selection, plan, PoC intent) from each phase's conversation, so that the session JSON contains actionable data for export, `sofia dev`, and progress tracking — regardless of whether the LLM uses JSON blocks, markdown tables, or prose.
+
+**Why this priority**: This is the root cause of BUG-002 and BUG-004. Without reliable extraction, the entire downstream pipeline (export, dev, status) is broken. The LLM produces high-quality content, but it's lost because the extractors are too rigid.
+
+**Independent Test**: Run a workshop session through all phases with real LLM output, then verify that `session.ideas`, `session.evaluation`, `session.selection`, `session.plan`, and `session.poc` are all populated in the session JSON. Alternatively, feed recorded LLM outputs from the Zava assessment into the extractors and verify structured data is produced.
+
+**Acceptance Scenarios**:
+
+1. **Given** the Ideate phase produces ideas in markdown format (titles, descriptions, bullet points) without a JSON code block, **When** the phase completes, **Then** `session.ideas` contains at least 3 extracted `IdeaCard` entries with title and description populated.
+2. **Given** the Design phase produces a feasibility-value scoring table in markdown, **When** the phase completes, **Then** `session.evaluation` contains scored ideas matching the LLM's output.
+3. **Given** the Select phase produces a recommendation with rationale in prose, **When** the user confirms the selection, **Then** `session.selection` contains the selected idea ID, rationale, and `confirmedByUser: true`.
+4. **Given** the Plan phase produces milestones in a numbered list or markdown structure, **When** the phase completes, **Then** `session.plan` contains milestone entries with titles and descriptions.
+5. **Given** the Develop boundary phase captures PoC intent in prose, **When** the phase completes, **Then** `session.poc` contains target stack, key scenarios, and constraints.
+6. **Given** the LLM response includes a valid JSON code block (current behavior), **When** the extractor runs, **Then** the JSON block extraction still works (backward compatibility preserved).
+
+---
+
+### User Story 2 — Web search is available when `.env` is loaded (Priority: P1)
+
+As a facilitator starting a workshop session, I want sofIA's web search to work when my `.env` file contains the correct Foundry credentials, regardless of module import ordering, so that the Discover phase can search the web for my company and industry context.
+
+**Why this priority**: BUG-001 means web search silently fails in any execution path where `envLoader.ts` hasn't run before `webSearch.ts` is first evaluated. This affects both the CLI and programmatic test paths.
+
+**Independent Test**: Set `FOUNDRY_PROJECT_ENDPOINT` and `FOUNDRY_MODEL_DEPLOYMENT_NAME` in `.env` only (not in shell env), start sofIA, and verify `isWebSearchConfigured()` returns true and the Discover phase offers web search.
+
+**Acceptance Scenarios**:
+
+1. **Given** `.env` contains valid Foundry credentials and `process.env` does not have them pre-set, **When** the CLI starts and calls `isWebSearchConfigured()`, **Then** it returns `true`.
+2. **Given** `.env` is absent and Foundry env vars are not set, **When** the CLI starts and calls `isWebSearchConfigured()`, **Then** it returns `false` (no error, no crash).
+3. **Given** Foundry env vars are set in the shell environment (not in `.env`), **When** the CLI starts, **Then** `isWebSearchConfigured()` returns `true` regardless of `.env` loading order.
+
+---
+
+### User Story 3 — Workshop phases use MCP tools for enrichment (Priority: P2)
+
+As a facilitator running a workshop, I want sofIA to use web search during Discover, and Context7/Azure MCP during Design and Plan, so that the workshop output is grounded in real-world data and current documentation rather than relying solely on the LLM's training data.
+
+**Why this priority**: BUG-005 prevents all MCP-driven enrichment in the workshop flow. While the workshop works without tools (graceful degradation is solid), the quality gap is significant — enriched sessions produce more relevant ideas and better-grounded architecture recommendations.
+
+**Independent Test**: Run a workshop session with MCP servers configured, verify that: (a) Discover phase calls `web.search` and stores results, (b) Design phase queries Context7 for library docs when ideas reference specific technologies, (c) Plan phase queries Azure MCP when the plan references Azure services.
+
+**Acceptance Scenarios**:
+
+1. **Given** web search is configured and the Discover phase has collected business context, **When** the enrichment hook runs, **Then** web search is called with company and industry queries and results are stored in `session.discovery.enrichment`.
+2. **Given** Context7 is available and the Design phase discusses ideas referencing npm packages, **When** the Design phase handler retrieves references, **Then** Context7 is queried for relevant library documentation.
+3. **Given** Azure MCP is available and the Plan phase references Azure services (e.g., Azure Functions, Cosmos DB), **When** the Plan phase handler builds the system prompt or post-processes, **Then** Azure architecture guidance is fetched and included in the LLM's context.
+4. **Given** all MCP tools are unavailable, **When** the workshop runs, **Then** every phase still completes successfully (existing graceful degradation preserved).
+
+---
+
+### User Story 4 — Long sessions don't cause timeouts (Priority: P2)
+
+As a facilitator completing a multi-phase workshop with rich conversations, I want sofIA to manage its context window so that later phases (Select, Plan, Develop) don't time out due to accumulated conversation history from earlier phases.
+
+**Why this priority**: BUG-003 caused the Select phase to time out completely, losing all user progress for that phase. In a real workshop, this would be frustrating and would undermine trust in the tool.
+
+**Independent Test**: Run a 6-phase workshop session with at least 10 turns per phase. Verify that the Select and Plan phases complete successfully without SDK timeouts, and that prior phase context is available to the LLM in summarized form.
+
+**Acceptance Scenarios**:
+
+1. **Given** a session with 30+ turns across Discover, Ideate, and Design phases, **When** the Select phase starts, **Then** it completes within 120 seconds without a timeout error.
+2. **Given** a session starting a new phase after multiple completed phases, **When** the ConversationLoop builds the context, **Then** prior phase turns are summarized (not included verbatim) in the system prompt to reduce context size.
+3. **Given** the summarized context from prior phases, **When** the LLM generates phase output, **Then** it can reference key decisions from earlier phases (business context, selected ideas) accurately despite summarization.
+
+---
+
+### User Story 5 — Export includes all phases with conversation fallback (Priority: P2)
+
+As a facilitator exporting workshop results, I want `sofia export` to produce markdown files for every phase that had conversations — even if structured artifacts weren't extracted — so that the full workshop output is preserved for review.
+
+**Why this priority**: BUG-004 means that only the Discover phase (which has `businessContext` populated) generates an export file. The other 5 phases' conversations are lost from the export despite containing rich, useful content. A workshop facilitator needs all phases represented.
+
+**Independent Test**: Run `sofia export` on a session that completed all phases but has null structured fields for Ideate through Develop. Verify that export files are generated for all phases using conversation turn fallback.
+
+**Acceptance Scenarios**:
+
+1. **Given** a session with Ideate conversation turns but no extracted `session.ideas`, **When** `sofia export` runs, **Then** an `ideate.md` file is generated containing the conversation turns for the Ideate phase.
+2. **Given** a session with all 6 phases completed and conversation data, **When** `sofia export` runs, **Then** the export directory contains `discover.md`, `ideate.md`, `design.md`, `select.md`, `plan.md`, and `develop.md`.
+3. **Given** a session with both structured data and conversation turns for a phase, **When** `sofia export` runs, **Then** the structured data is rendered first, followed by the conversation.
+4. **Given** `summary.json`, **When** generated, **Then** it lists all exported phase files and includes highlights from every populated phase.
+
+---
+
+### Edge Cases
+
+- What if the LLM produces a JSON block that partially matches the schema (e.g., missing one required field)? The summarization call should still produce a valid extraction; the original partial JSON should be logged for debugging.
+- What if the summarization LLM call itself fails? Fall back to the current extraction behavior (JSON block parsing) and log a warning. The phase should not be blocked by a failed summarization attempt.
+- What if context summarization loses critical details (e.g., a specific technology choice from the Plan)? The system should preserve key fields (business context, selected idea, plan milestones) verbatim, and only summarize conversation turns.
+- What if the user's session has 0 turns for a phase (e.g., the Select timeout scenario)? The export phase file should still be generated with a note that the phase had no conversation content.
+- What if multiple JSON blocks exist in a single LLM response? The extractor should try all blocks against the expected schema, not just the first one.
+- What if the LLM drifts into the next phase's content before the decision gate? The system prompt should enforce phase boundaries explicitly.
+- What if the Select phase still times out even after context summarization? The system should have a secondary fallback (minimal-context retry or user-directed selection).
+- What if the Zava assessment live test provides more or fewer inputs than the LLM expects? The test harness should detect input exhaustion and signal completion gracefully rather than silently consuming inputs across phase boundaries. (Test infrastructure concern — not a production code issue.)
+
+## Requirements _(mandatory)_
+
+### Functional Requirements
+
+#### Phase Extraction Hardening (BUG-002)
+
+- **FR-001**: At the end of each phase (when the conversation loop exits), the system MUST make a dedicated "summarization" LLM call that asks the model to output the phase's structured data as a JSON code block, using the full conversation from that phase as context.
+- **FR-002**: The summarization call MUST use a phase-specific prompt that instructs the LLM to extract the exact JSON shape expected by the schema (e.g., `IdeaCard[]`, `IdeaEvaluation`, `SelectedIdea`, `ImplementationPlan`, `PocDevelopmentState`).
+- **FR-003**: The existing `extractJsonBlock()` → schema parsing pipeline MUST be preserved as the primary extraction path during the conversation. The summarization call is an additional fallback that runs once at phase end.
+- **FR-004**: If the summarization call produces valid structured data and the session field is still null (i.e., per-turn extraction didn't capture it), the system MUST populate the session field from the summarization result.
+- **FR-005**: If the summarization call fails or returns invalid data, the system MUST log a warning and continue without blocking the phase transition. Existing per-turn extraction results (if any) MUST be preserved.
+- **FR-006**: The summarization call MUST be implemented as a method on `ConversationLoop` (or a utility invoked by it) so that all phases benefit without duplicating logic per handler.
+- **FR-007**: The existing `extractJsonBlock()` function MUST be enhanced to try multiple JSON blocks in a response (not just the first match) and return the first one that validates against the expected schema.
+- **FR-007a**: The Design phase summarization prompt (FR-002) MUST also request a Mermaid architecture diagram alongside the structured evaluation JSON, fulfilling spec 001 FR-030. The diagram MUST be stored in the session (as part of `evaluation` or a dedicated field) for export.
+
+#### Phase Boundary Enforcement
+
+- **FR-007b**: The system prompt for each phase MUST include an explicit instruction prohibiting the LLM from introducing or transitioning to the next phase. The instruction MUST state: "You are in the [Phase] phase. Do NOT introduce or begin the next phase. The user will be offered a decision gate when this phase is complete."
+- **FR-007c**: The `ConversationLoop` (or phase handler `buildSystemPrompt`) MUST inject the phase-boundary instruction automatically for all phases, without requiring per-handler duplication.
+
+#### Lazy Web Search Configuration (BUG-001)
+
+- **FR-008**: `isWebSearchConfigured()` MUST evaluate `process.env` at call time (lazy), not at module load time. The function MUST NOT cache or memoize the result of the environment variable check.
+- **FR-009**: All callers of `isWebSearchConfigured()` MUST continue to call it as a function. No API signature changes required.
+- **FR-010**: The CLI startup sequence MUST ensure `loadEnvFile()` runs before any code path checks `isWebSearchConfigured()`. This ordering MUST be enforced by calling `loadEnvFile()` at the top of the `workshopCommand()` and `developCommand()` entry points.
+
+#### MCP Tool Wiring in Workshop Flow (BUG-005)
+
+- **FR-011**: `workshopCommand.ts` MUST create an `McpManager` instance at startup (if MCP configuration exists in `.vscode/mcp.json` or equivalent).
+- **FR-012**: `workshopCommand.ts` MUST create a `WebSearchClient` from the configured Foundry agent credentials (if web search is configured) and pass it to the Discover phase handler via `DiscoverHandlerConfig`.
+- **FR-012a**: `workshopCommand.ts` MUST pass the `McpManager` to the Discover phase handler so that it can check WorkIQ availability. When WorkIQ is available, the Discover handler MUST prompt the user for explicit consent before querying WorkIQ (per spec 001 FR-020 and spec 003 US4). WorkIQ insights MUST be stored in `session.discovery.enrichment.workiqInsights`.
+- **FR-013**: The Design phase handler MUST accept optional MCP configuration and use Context7 to fetch library documentation for technologies referenced in the ideas, when available.
+- **FR-014**: The Plan phase handler MUST accept optional MCP configuration and use Azure MCP / Microsoft Learn to fetch architecture guidance for Azure services referenced in the plan, when available.
+- **FR-015**: All MCP tool calls from workshop phase handlers MUST degrade gracefully — if a tool is unavailable or errors, the phase continues with LLM-only output.
+
+#### Context Window Management (BUG-003)
+
+- **FR-016**: When starting a new phase, the `ConversationLoop` MUST NOT include raw conversation turns from previous phases in the system prompt. Instead, it MUST include a summarized context block.
+- **FR-017**: The summarized context MUST preserve: business context, topic, key decisions, selected idea (if applicable), plan milestones (if applicable), discovery enrichment data (web search results, WorkIQ insights — if populated), and any other structured session fields already extracted.
+- **FR-018**: Only conversation turns from the current phase MUST be included in the session history injection (for resume scenarios). Turns from prior phases MUST be summarized.
+- **FR-019**: The system SHOULD use the SDK's `infiniteSessions` configuration for long-running sessions as an additional protection against context exhaustion.
+- **FR-019a**: If a phase times out even after context summarization (FR-016–FR-018), the system MUST retry once with a minimal context payload containing only the structured session fields (no conversation turns at all). If the retry also fails, the system MUST fall back to asking the user to manually confirm or provide the expected output (e.g., for Select: present the top-ranked idea from Design and ask the user to confirm).
+
+#### Export Completeness (BUG-004)
+
+- **FR-020**: The export writer MUST generate a phase markdown file for any phase that has conversation turns in the session, even if the structured session field for that phase is null.
+- **FR-021**: When generating a phase file without structured data, the export writer MUST include the conversation turns formatted as a readable transcript.
+- **FR-022**: When generating a phase file with both structured data and conversation turns, the export writer MUST include the structured data first, then the conversation as a "Conversation" section.
+- **FR-023**: `summary.json` MUST list all generated phase files, not just those with structured data.
+- **FR-024**: `summary.json` highlights MUST include at least one highlight per phase that had conversation turns, derived from the conversation content or structured data.
+
+### Key Entities
+
+- **PhaseSummarizationRequest**: A structured request to the LLM at phase end, containing the full conversation transcript for the phase and a phase-specific extraction prompt. Produces the expected schema (e.g., `IdeaCard[]`, `ImplementationPlan`).
+- **SummarizedPhaseContext**: A compact representation of a completed phase's key outputs, used to inject prior-phase context into subsequent phases without including raw turns.
+- **McpWorkshopConfig**: Configuration object passed from `workshopCommand.ts` to phase handler factories, containing optional `McpManager` and `WebSearchClient` references for tool-based enrichment.
+
+## Success Criteria _(mandatory)_
+
+### Measurable Outcomes
+
+- **SC-001**: A full 6-phase workshop session produces non-null values for at least `session.businessContext`, `session.ideas`, `session.selection`, and `session.plan` in 90%+ of properly configured runs.
+- **SC-002**: The `sofia export` command generates markdown files for all 6 phases (Discover through Develop) when the session has conversation data for all phases, regardless of structured data availability.
+- **SC-003**: `isWebSearchConfigured()` returns the correct value when `.env` is the sole source of Foundry credentials, verified by a unit test that sets env vars after module import.
+- **SC-004**: A workshop session with 40+ turns across 4+ phases completes the Select and Plan phases without SDK timeout errors.
+- **SC-005**: When web search and at least one MCP server (Context7 or Azure) are configured, the workshop session uses those tools during the appropriate phases and stores the results in the session.
+- **SC-006**: The Zava Industries assessment test (or equivalent) scores at least 75% on testable checks (up from 53%) after these fixes are applied.
+
+## Assumptions
+
+- The Copilot SDK's `sendAndWait()` method can handle a 2-turn summarization call (system prompt + phase transcript) within the existing 120-second timeout.
+- The LLM is capable of producing valid JSON that matches the session schema when given explicit instructions and the full conversation transcript — this has been verified in the Discover phase where `businessContext` extraction already works.
+- The existing test suite (709 unit + 99 integration tests) continues to pass after these changes, as the fixes are backward-compatible with existing behavior.
+- MCP server configurations in `.vscode/mcp.json` are readable by `McpManager` — this was already implemented in Feature 003.
+- The `infiniteSessions` SDK feature (documented in `copilotClient.ts`) is stable enough for production use.
+
+## Dependencies
+
+- **Feature 001**: Session schema, ConversationLoop, phase handlers, export writer
+- **Feature 003**: McpManager, MCP transport layer, DiscoveryEnricher, web search client
+- **Feature 005**: Foundry deployment, `.env` output, `isWebSearchConfigured()`
+
+## Out of Scope
+
+- **Prose-based extraction without summarization call**: Building NLP-based extractors that parse markdown tables and prose directly is complex and fragile. The summarization call approach is simpler and leverages the LLM's own ability to restructure its output.
+- **Automatic retry for the Select timeout**: While BUG-003 caused a timeout, the root fix is context management (FR-016), not retry logic. The existing retry infrastructure (FR-050 from spec 001) handles transient failures.
+- **Multi-language template support for PoC generation**: Template changes belong in Feature 004.
+- **PTY-based E2E test automation**: The Zava assessment test is a programmatic live test. Full PTY-based interactive testing is deferred to Feature 004.

package/specs/006-workshop-extraction-fixes/tasks.md

@@ -0,0 +1,316 @@
+# Tasks: Workshop Phase Extraction & Tool Wiring Fixes
+
+**Input**: Design documents from `/specs/006-workshop-extraction-fixes/`  
+**Prerequisites**: plan.md, spec.md, research.md, data-model.md, contracts/
+
+**Tests**: Tests are REQUIRED for new behavior (Red → Green → Review). Test tasks are included for each user story and MUST be written first.
+
+**Organization**: Tasks are grouped by user story (from spec.md) to enable independent implementation and testing.
+
+## Format: `[ID] [P?] [Story?] Description`
+
+- **[P]**: Can run in parallel (different files, no dependencies on incomplete tasks)
+- **[Story]**: Which user story (US1–US5) this task belongs to
+
+> **Note**: Implementation phases are ordered for efficiency (US2 before US1 because it's a quick fix).
+> This differs from spec story numbering where US1=Extraction, US2=WebSearch.
+
+---
+
+## Phase 1: Setup
+
+**Purpose**: Project structure for new modules; no behavior changes yet
+
+- [x] T001 Create empty module file src/loop/phaseSummarizer.ts with JSDoc header and type-only imports
+- [x] T002 [P] Create empty module file src/phases/contextSummarizer.ts with JSDoc header and type-only imports
+- [x] T003 [P] Create prompt directory src/prompts/summarize/ with placeholder README
+- [x] T004 [P] Update src/prompts/promptLoader.ts to support loading summarization prompts from `summarize/` subdirectory
+- [x] T005 Verify `npm run typecheck` and `npm run lint` pass with empty modules
+
+---
+
+## Phase 2: Foundational (Blocking Prerequisites)
+
+**Purpose**: Multi-JSON-block extraction and phase boundary enforcement — used by multiple user stories
+
+**⚠️ CRITICAL**: US1 (extraction) and US4 (context) depend on these foundational changes
+
+### Tests (REQUIRED — write first, must FAIL) ⚠️
+
+- [x] T006 [P] Add failing tests for `extractAllJsonBlocks()` in tests/unit/phases/phaseExtractors.spec.ts — test with 0, 1, 2, and 3 JSON blocks in a single response
+- [x] T007 [P] Add failing tests for `extractJsonBlockForSchema()` in tests/unit/phases/phaseExtractors.spec.ts — test with multiple blocks where only the second matches the schema
+- [x] T008 [P] Add failing test for phase boundary injection in tests/unit/phases/phaseHandlers.spec.ts — verify system prompt contains "Do NOT introduce or begin the next phase"
+
+### Implementation
+
+- [x] T009 Implement `extractAllJsonBlocks()` in src/phases/phaseExtractors.ts — use `/g` flag for fenced blocks, bracket-depth counter for raw JSON (FR-007)
+- [x] T010 Implement `extractJsonBlockForSchema<T>()` in src/phases/phaseExtractors.ts — try each block with `safeParse()`, return first valid match (FR-007)
+- [x] T011 [P] Inject phase-boundary instruction in ConversationLoop system prompt builder in src/loop/conversationLoop.ts (FR-007b, FR-007c)
+- [x] T012 Run `npm run test:unit` — T006, T007, T008 must now PASS; all existing 709 tests must still PASS
+
+**Checkpoint**: Foundational extraction + boundary enforcement ready. User story work can begin.
+
+---
+
+## Phase 3: User Story 2 — Lazy Web Search Config (Priority: P1) 🎯 MVP-1
+
+**Goal**: `isWebSearchConfigured()` returns correct value regardless of `.env` loading order (BUG-001)
+
+**Independent Test**: Set Foundry vars in `.env` only, import `webSearch.ts`, then call `isWebSearchConfigured()` and expect `true`.
+
+### Tests for US2 (REQUIRED — write first, must FAIL) ⚠️
+
+- [x] T013 [P] [US2] Add failing test in tests/unit/mcp/webSearch.spec.ts — set env vars AFTER module import, verify `isWebSearchConfigured()` returns true
+- [x] T014 [P] [US2] Add failing test in tests/unit/mcp/webSearch.spec.ts — verify returns false when vars absent
+- [x] T015 [P] [US2] Add failing test in tests/unit/cli/workshopCommand.spec.ts — verify `loadEnvFile()` is called before workshop logic starts
+
+### Implementation for US2
+
+- [x] T016 [US2] Verify `isWebSearchConfigured()` in src/mcp/webSearch.ts reads `process.env` at call time with no caching (FR-008, FR-009)
+- [x] T017 [US2] Add `loadEnvFile()` call at top of `workshopCommand()` in src/cli/workshopCommand.ts (FR-010)
+- [x] T018 [P] [US2] Add `loadEnvFile()` call at top of `developCommand()` in src/cli/developCommand.ts (FR-010)
+- [x] T019 [US2] Run `npm run test:unit` — T013, T014, T015 must now PASS
+
+**Checkpoint**: Web search configuration works reliably. Can be verified in isolation.
+
+---
+
+## Phase 4: User Story 1 — Phase Data Extraction (Priority: P1) 🎯 MVP-2
+
+**Goal**: Structured artifacts (ideas, evaluation, selection, plan, poc) reliably extracted from every workshop phase via post-phase summarization call (BUG-002)
+
+**Independent Test**: Feed recorded Zava assessment conversations into the summarization pipeline, verify all session fields populated.
+
+### Tests for US1 (REQUIRED — write first, must FAIL) ⚠️
+
+- [x] T020 [P] [US1] Add failing test for `phaseSummarize()` in tests/unit/loop/phaseSummarizer.spec.ts — with fake client returning JSON block, verify session field populated
+- [x] T021 [P] [US1] Add failing test for `phaseSummarize()` in tests/unit/loop/phaseSummarizer.spec.ts — with fake client returning invalid response, verify session unchanged (no crash)
+- [x] T022 [P] [US1] Add failing test for `phaseSummarize()` in tests/unit/loop/phaseSummarizer.spec.ts — field already populated, verify summarization skipped (no-op)
+- [x] T023 [P] [US1] Add failing test for Ideate summarization prompt in tests/unit/loop/phaseSummarizer.spec.ts — verify IdeaCard[] extracted from LLM summary response
+- [x] T024 [P] [US1] Add failing test for Design summarization + Mermaid diagram extraction in tests/unit/loop/phaseSummarizer.spec.ts (FR-007a)
+- [x] T025 [P] [US1] Add failing integration test in tests/integration/summarizationFlow.spec.ts — full pipeline: ConversationLoop → phaseSummarize → session updated
+
+### Implementation for US1
+
+- [x] T026 [US1] Create summarization prompt src/prompts/summarize/ideate-summary.md — IdeaCard[] schema shape + extraction instructions (FR-002)
+- [x] T027 [P] [US1] Create summarization prompt src/prompts/summarize/design-summary.md — IdeaEvaluation schema + Mermaid diagram request (FR-002, FR-007a)
+- [x] T028 [P] [US1] Create summarization prompt src/prompts/summarize/select-summary.md — SelectedIdea schema shape (FR-002)
+- [x] T029 [P] [US1] Create summarization prompt src/prompts/summarize/plan-summary.md — ImplementationPlan schema shape (FR-002)
+- [x] T030 [P] [US1] Create summarization prompt src/prompts/summarize/develop-summary.md — PocDevelopmentState schema shape (FR-002)
+- [x] T031 [US1] Implement `phaseSummarize()` in src/loop/phaseSummarizer.ts — create new session, send transcript, extract with handler (FR-001, FR-003, FR-004, FR-005, FR-006). The Discover phase MAY skip this call if `businessContext` is already populated.
+- [x] T032 [US1] Implement `needsSummarization()` in src/loop/phaseSummarizer.ts — check if phase's session field is null
+- [x] T033 [US1] Implement `buildPhaseTranscript()` in src/loop/phaseSummarizer.ts — concatenate user+assistant turns for the phase
+- [x] T034 [US1] Implement Mermaid diagram extraction in Design summarization path in src/loop/phaseSummarizer.ts — extract `mermaid` block from summarization response and store in session `evaluation.architectureDiagram` (FR-007a)
+- [x] T035 [US1] Hook `phaseSummarize()` into ConversationLoop.run() after while loop exits, before return, in src/loop/conversationLoop.ts (FR-006)
+- [x] T036 [US1] Run `npm run test:unit && npm run test:integration` — T020–T025 must PASS; all existing tests must still PASS
+
+**Checkpoint**: All phases extract structured data via summarization fallback. Session fields populated.
+
+---
+
+## Phase 5: User Story 4 — Context Window Management (Priority: P2)
+
+**Goal**: Later phases (Select, Plan, Develop) don't time out due to accumulated conversation history (BUG-003)
+
+**Independent Test**: Build a session with 40+ turns, start Select phase, verify no timeout and summarized context in prompt.
+
+### Tests for US4 (REQUIRED — write first, must FAIL) ⚠️
+
+- [x] T037 [P] [US4] Add failing test for `buildSummarizedContext()` in tests/unit/phases/contextSummarizer.spec.ts — with full session, verify all fields projected
+- [x] T038 [P] [US4] Add failing test for `buildSummarizedContext()` in tests/unit/phases/contextSummarizer.spec.ts — with null fields, verify graceful omission
+- [x] T039 [P] [US4] Add failing test for `renderSummarizedContext()` in tests/unit/phases/contextSummarizer.spec.ts — verify markdown output format
+- [x] T040 [P] [US4] Add failing test in tests/unit/phases/phaseHandlers.spec.ts — verify Ideate handler uses renderSummarizedContext (not ad-hoc context)
+- [x] T041 [P] [US4] Add failing test for ConversationLoop infiniteSessions forwarding in tests/unit/loop/conversationLoop.spec.ts
+- [x] T042 [P] [US4] Add failing test for timeout-retry fallback (FR-019a) in tests/unit/loop/conversationLoop.spec.ts — on timeout, retry with minimal context
+- [x] T043 [P] [US4] Add failing test for user-directed fallback (FR-019a) in tests/unit/loop/conversationLoop.spec.ts — on second timeout, ask user for manual input
+
+### Implementation for US4
+
+- [x] T044 [US4] Implement `buildSummarizedContext()` in src/phases/contextSummarizer.ts — project all structured session fields including discoveryEnrichment (FR-016, FR-017)
+- [x] T045 [US4] Implement `renderSummarizedContext()` in src/phases/contextSummarizer.ts — render as compact markdown section (FR-017)
+- [x] T046 [US4] Replace ad-hoc context blocks in Ideate handler's `buildSystemPrompt()` with `renderSummarizedContext()` in src/phases/phaseHandlers.ts (FR-016)
+- [x] T047 [P] [US4] Replace ad-hoc context blocks in Design handler's `buildSystemPrompt()` with `renderSummarizedContext()` in src/phases/phaseHandlers.ts (FR-016)
+- [x] T048 [P] [US4] Replace ad-hoc context blocks in Select handler's `buildSystemPrompt()` with `renderSummarizedContext()` in src/phases/phaseHandlers.ts (FR-016)
+- [x] T049 [P] [US4] Replace ad-hoc context blocks in Plan handler's `buildSystemPrompt()` with `renderSummarizedContext()` in src/phases/phaseHandlers.ts (FR-016)
+- [x] T050 [P] [US4] Replace ad-hoc context blocks in Develop handler's `buildSystemPrompt()` with `renderSummarizedContext()` in src/phases/phaseHandlers.ts (FR-016)
+- [x] T051 [US4] Verify ConversationLoop.run() only injects current-phase turns (not prior phases) in system prompt history block in src/loop/conversationLoop.ts (FR-018 — already implemented, add regression test)
+- [x] T052 [US4] Add `infiniteSessions` option to `ConversationLoopOptions` and forward to `createSession()` in src/loop/conversationLoop.ts (FR-019)
+- [x] T053 [US4] Implement minimal-context retry on timeout in ConversationLoop — on `sendAndWait` timeout, retry with only structured session fields and no conversation turns (FR-019a)
+- [x] T054 [US4] Implement user-directed fallback in ConversationLoop — on second timeout after retry, present the best available answer to user and ask for manual confirmation (FR-019a)
+- [x] T055 [US4] Pass `infiniteSessions: { backgroundCompactionThreshold: 0.7, bufferExhaustionThreshold: 0.9 }` from workshopCommand.ts to ConversationLoop (FR-019)
+- [x] T056 [US4] Run `npm run test:unit` — T037–T043 must PASS; all existing tests must still PASS
+
+**Checkpoint**: Select/Plan phases complete without timeout. Context is compact and accurate.
+
+---
+
+## Phase 6: User Story 3 — MCP Tool Wiring (Priority: P2)
+
+**Goal**: Workshop phases use web search, WorkIQ, Context7, and Azure MCP for enrichment (BUG-005)
+
+**Independent Test**: With MCP configured, run Discover and verify web search results stored; run Design and verify Context7 queried.
+
+### Tests for US3 (REQUIRED — write first, must FAIL) ⚠️
+
+- [x] T057 [P] [US3] Add failing test in tests/unit/cli/workshopCommand.spec.ts — verify McpManager created from .vscode/mcp.json
+- [x] T058 [P] [US3] Add failing test in tests/unit/cli/workshopCommand.spec.ts — verify WebSearchClient created when configured and passed to Discover handler
+- [x] T059 [P] [US3] Add failing test in tests/unit/cli/workshopCommand.spec.ts — verify McpManager passed to Discover handler for WorkIQ consent flow (FR-012a)
+- [x] T060 [P] [US3] Add failing test in tests/unit/phases/phaseHandlers.spec.ts — verify Design handler queries Context7 via McpManager in postExtract (FR-013)
+- [x] T061 [P] [US3] Add failing test in tests/unit/phases/phaseHandlers.spec.ts — verify Plan handler queries Azure MCP via McpManager in postExtract (FR-014)
+- [x] T062 [P] [US3] Add failing test in tests/unit/phases/phaseHandlers.spec.ts — verify Design handler degrades gracefully when Context7 unavailable (FR-015)
+
+### Implementation for US3
+
+- [x] T063 [US3] Extend `PhaseHandlerConfig` with `mcpManager?: McpManager` and `webSearchClient?: WebSearchClient` in src/phases/phaseHandlers.ts (FR-011)
+- [x] T064 [US3] Create `McpManager` in `workshopCommandInner()` from `.vscode/mcp.json` via `loadMcpConfig()` in src/cli/workshopCommand.ts (FR-011)
+- [x] T065 [US3] Create `WebSearchClient` in `workshopCommandInner()` when `isWebSearchConfigured()` returns true in src/cli/workshopCommand.ts (FR-012)
+- [x] T066 [US3] Pass `mcpManager` + `webSearchClient` to Discover handler via `PhaseHandlerConfig.discover` in src/cli/workshopCommand.ts (FR-012, FR-012a — verify existing WorkIQ consent flow activates when McpManager is wired)
+- [x] T067 [US3] Pass `mcpManager` to all phase handler calls via `PhaseHandlerConfig` in src/cli/workshopCommand.ts
+- [x] T068 [US3] Add `postExtract` hook to Design handler — query Context7 for technologies in `session.ideas` in src/phases/phaseHandlers.ts (FR-013)
+- [x] T069 [US3] Add `postExtract` hook to Plan handler — query Azure MCP for services in `session.plan.architectureNotes` in src/phases/phaseHandlers.ts (FR-014)
+- [x] T070 [US3] Wrap all MCP calls in try/catch for graceful degradation in src/phases/phaseHandlers.ts (FR-015)
+- [x] T071 [US3] Run `npm run test:unit` — T057–T062 must PASS; all existing tests must still PASS
+
+**Checkpoint**: MCP tools wired and operational. Enrichment flows working with graceful degradation.
+
+---
+
+## Phase 7: User Story 5 — Export Completeness (Priority: P2)
+
+**Goal**: `sofia export` produces markdown files for all phases with conversation data, even without structured artifacts (BUG-004)
+
+**Independent Test**: Export a session with null structured fields but 48 conversation turns — verify 6 markdown files generated.
+
+### Tests for US5 (REQUIRED — write first, must FAIL) ⚠️
+
+- [x] T072 [P] [US5] Add failing test in tests/unit/sessions/exportWriter.spec.ts — Ideate export with null `session.ideas` but conversation turns produces ideate.md
+- [x] T073 [P] [US5] Add failing test in tests/unit/sessions/exportWriter.spec.ts — Design export with null `session.evaluation` but turns produces design.md
+- [x] T074 [P] [US5] Add failing test in tests/unit/sessions/exportWriter.spec.ts — export with both structured data + turns renders structured first then conversation
+- [x] T075 [P] [US5] Add failing test in tests/unit/sessions/exportWriter.spec.ts — summary.json lists all 6 phase files when all phases have turns
+- [x] T076 [P] [US5] Add failing test in tests/unit/sessions/exportWriter.spec.ts — summary.json highlights include one entry per phase with turns
+- [x] T077 [P] [US5] Add failing integration test in tests/integration/exportFallbackFlow.spec.ts — full export pipeline with null structured data
+
+### Implementation for US5
+
+- [x] T078 [US5] Refactor `generateIdeateMarkdown()` in src/sessions/exportWriter.ts — remove early return null; add conversation turn fallback (FR-020, FR-021, FR-022)
+- [x] T079 [P] [US5] Refactor `generateDesignMarkdown()` in src/sessions/exportWriter.ts — same pattern (FR-020, FR-021, FR-022)
+- [x] T080 [P] [US5] Refactor `generateSelectMarkdown()` in src/sessions/exportWriter.ts — same pattern (FR-020, FR-021, FR-022)
+- [x] T081 [P] [US5] Refactor `generatePlanMarkdown()` in src/sessions/exportWriter.ts — same pattern (FR-020, FR-021, FR-022)
+- [x] T082 [P] [US5] Refactor `generateDevelopMarkdown()` in src/sessions/exportWriter.ts — same pattern (FR-020, FR-021, FR-022)
+- [x] T083 [US5] Update `exportSession()` in src/sessions/exportWriter.ts — summary.json lists all generated files (FR-023)
+- [x] T084 [US5] Update highlight generation in src/sessions/exportWriter.ts — include one highlight per phase with turns, fallback to first assistant turn opening (FR-024)
+- [x] T085 [US5] Run `npm run test:unit && npm run test:integration` — T072–T077 must PASS; all existing tests must still PASS
+
+**Checkpoint**: Export produces complete artifacts for all 6 phases. summary.json includes all files and highlights.
+
+---
+
+## Phase 8: Polish & Cross-Cutting Concerns
+
+**Purpose**: Final validation, regression testing, cleanup
+
+- [x] T086 [P] Update build assets script in package.json to copy `src/prompts/summarize/*.md` to `dist/src/prompts/summarize/`
+- [x] T087 [P] Run `npm run typecheck` — zero errors
+- [x] T088 [P] Run `npm run lint` — zero errors (fix any import ordering issues)
+- [x] T089 Run full test suite: `npm run test:unit && npm run test:integration && npm run test:e2e`
+- [x] T090 Update Zava assessment test in tests/live/zavaFullWorkshop.spec.ts — relax assertion on `session.ideas` (now expected to pass), add assertions for extraction, export completeness
+- [x] T091 Add failure/recovery E2E scenario to Zava live test in tests/live/zavaFullWorkshop.spec.ts — simulate a phase timeout and verify recovery fallback activates (FR-019a, Constitution Principle VI)
+- [x] T092 Run quickstart.md validation — verify all file paths and commands in quickstart.md are accurate
+
+---
+
+## Dependencies & Execution Order
+
+### Phase Dependencies
+
+- **Phase 1 (Setup)**: No dependencies — start immediately
+- **Phase 2 (Foundational)**: Depends on Phase 1 — BLOCKS all user stories
+- **Phase 3 (US2 — Web Search)**: Depends on Phase 2 — independent of other stories
+- **Phase 4 (US1 — Extraction)**: Depends on Phase 2 — independent of other stories
+- **Phase 5 (US4 — Context)**: Depends on Phase 2 — independent of other stories
+- **Phase 6 (US3 — MCP Wiring)**: Depends on Phase 3 (needs lazy web search) — can parallel with US1/US4
+- **Phase 7 (US5 — Export)**: Depends on Phase 2 — independent of other stories (benefits from US1 but works without)
+- **Phase 8 (Polish)**: Depends on all user stories being complete
+
+### User Story Dependencies
+
+| Story                 | Can Start After | Parallel With      | Notes                                                    |
+| --------------------- | --------------- | ------------------ | -------------------------------------------------------- |
+| US2 (P1 — Web Search) | Phase 2         | US1, US4, US5      | Quick win — 3 FRs, ~4 tasks                              |
+| US1 (P1 — Extraction) | Phase 2         | US2, US4, US5      | Largest story — 7 FRs, ~16 tasks                         |
+| US4 (P2 — Context)    | Phase 2         | US1, US2, US5      | Modifies phaseHandlers shared with US3                   |
+| US3 (P2 — MCP)        | Phase 3 (US2)   | US1, US4, US5      | Needs web search from US2; shares phaseHandlers with US4 |
+| US5 (P2 — Export)     | Phase 2         | US1, US2, US3, US4 | Standalone module; benefits from US1 but works without   |
+
+### Within Each User Story
+
+1. Tests written first — MUST FAIL before implementation
+2. Implementation tasks in dependency order
+3. Green checkpoint: all tests pass
+4. Full suite verification before moving to next story
+
+### Parallel Opportunities
+
+```text
+After Phase 2 completes:
+
+  ┌──── US2 (Web Search) ─────┐
+  │                            │
+  │  ┌──── US1 (Extraction) ──┼──── US3 (MCP Wiring) ────┐
+  │  │                        │                            │
+  │  │  ┌── US4 (Context) ───┘                            │
+  │  │  │                                                  │
+  │  │  │  ┌── US5 (Export) ──────────────────────────────┘
+  │  │  │  │
+  ▼  ▼  ▼  ▼
+  Phase 8: Polish
+```
+
+---
+
+## Implementation Strategy
+
+### MVP First (US2 + US1 Only)
+
+1. Complete Phase 1: Setup
+2. Complete Phase 2: Foundational (multi-block extraction + phase boundaries)
+3. Complete Phase 3: US2 — Lazy Web Search Config (quick win, ~30 min)
+4. Complete Phase 4: US1 — Phase Data Extraction (summarization pipeline, ~2 hours)
+5. **STOP and VALIDATE**: Run Zava assessment test — extraction should now populate all fields
+
+### Incremental Delivery
+
+1. Setup + Foundational → Base ready
+2. US2 (Web Search) → Config bug fixed → Deploy
+3. US1 (Extraction) → Structured data flows → Deploy (major milestone)
+4. US4 (Context) → Timeout prevention → Deploy
+5. US3 (MCP Wiring) → Enrichment active → Deploy
+6. US5 (Export) → Complete exports → Deploy
+7. Polish → Full regression pass → Release
+
+### Suggested MVP Scope
+
+**US2 + US1** cover the two P1 stories and directly address:
+
+- BUG-001 (web search config)
+- BUG-002 (extraction failures)
+- Indirectly improves BUG-004 (export now has structured data to render)
+
+This combination targets the biggest score improvement in the Zava assessment.
+
+---
+
+## Summary
+
+| Metric                     | Value                                                 |
+| -------------------------- | ----------------------------------------------------- |
+| **Total tasks**            | 92                                                    |
+| **US1 (Extraction)**       | 17 tasks (incl. Mermaid extraction T034)              |
+| **US2 (Web Search)**       | 7 tasks                                               |
+| **US3 (MCP Wiring)**       | 15 tasks                                              |
+| **US4 (Context)**          | 20 tasks (incl. split timeout retry/fallback)         |
+| **US5 (Export)**           | 14 tasks                                              |
+| **Setup + Foundational**   | 12 tasks                                              |
+| **Polish**                 | 7 tasks (incl. failure/recovery E2E scenario)         |
+| **Parallel opportunities** | US2/US1/US4/US5 can all start after Phase 2 completes |
+| **MVP scope**              | US2 + US1 (24 tasks, addresses P1 stories)            |