@exaudeus/workrail 3.39.0 → 3.41.0

package/docs/ideas/backlog.md

@@ -5700,136 +5700,486 @@ Tested empirically today. This is what actually works, not what's specced.
 
 ---
 
-### Self-healing daemon: detect internal failures, kill, diagnose, fix, reboot, resume (Apr 18, 2026)
+### Autonomous feature development: scope → breakdown → parallel execution → merge (Apr 18, 2026)
 
-**The problem:** today if WorkRail's MCP connection drops or the daemon's internal tooling fails, agents continue running without enforcement -- producing unverified output that looks correct but bypassed all workflow gates. The user has no way to know this happened until they manually inspect session completion events.
+**The vision:** give WorkTrain a feature scope -- from a vague idea to a fully groomed ticket -- and it figures out the rest. Discovery if needed, design if needed, breakdown into parallel slices, execution across worktrees, context management across agents, bringing it all back together.
 
-**What happened today:** WorkRail MCP went down mid-session across ~10 concurrent agents. All sessions show INCOMPLETE (no `run_completed` event). Agents produced PRs, reviews, and merges -- two PRs landed on main -- without any confirmed workflow completion. Required manual audit after the fact.
+**The four pillars the user cares about:**
+1. **Autonomy** -- WorkTrain takes a scope and figures out the work breakdown without hand-holding
+2. **Quality** -- comes FROM autonomy + workflow enforcement + coordination. Each slice goes through the right phases.
+3. **Throughput** -- parallel slices across worktrees simultaneously. N agents working while you focus elsewhere.
+4. **Visibility** -- one coherent work unit you can track at a glance, not N unrelated sessions in a flat list.
 
-**What WorkTrain needs:**
+**The pipeline for a scope:**
 
-**1. Detect its own tooling failures**
-- Monitor whether `complete_step` / `continue_workflow` tool calls are succeeding or timing out
-- Detect when the WorkRail session store becomes unreachable
-- Detect when the MCP connection (for agents that use MCP-mode) is lost
-- Distinguish: "agent is thinking" vs "agent is stuck" vs "agent's tools are broken"
+```
+Input: "add GitHub polling support" (any level of definition -- idea to full spec)
+  │
+  ├── [if vague] ideation + spec authoring → output: BRD / acceptance criteria
+  ├── classify-task → taskComplexity, hasUI, touchesArchitecture, taskMaturity
+  ├── [if Medium/Large] discovery → context bundle, invariants, candidate files
+  ├── [if touchesArchitecture] design → candidates, review, selected approach
+  ├── breakdown → parallel slices with dependency graph
+  │     ├── Slice 1: types + schema         (worktree A)
+  │     ├── Slice 2: polling adapter        (worktree B, depends: 1)
+  │     ├── Slice 3: scheduler integration  (worktree C, depends: 2)
+  │     └── Slice 4: tests                 (worktree D, depends: 1-3)
+  ├── [parallel execution] each slice: implement → review → (fix if needed) → approved
+  ├── [serial integration] merge slices in dependency order, verify after each
+  └── [final] integration test → PR created → notification to user
+```
+
+**Context management across agents:**
+- Coordinator maintains a "work unit manifest": current phase, slice status, shared invariants, decisions made in design phase
+- Each spawned agent receives a context bundle: relevant portion of the manifest + files it needs + decisions from upstream phases
+- Agents don't rediscover what the coordinator already knows
+- After each agent completes, its findings update the manifest (new invariants found, scope changes, follow-up tickets)
+
+**Worktree coordination:**
+- Each slice gets its own worktree (already done via `--isolation worktree`)
+- Coordinator tracks which files each slice touches -- detects conflicts before they happen
+- Independent slices run in parallel; dependent slices queue automatically
+- Merge order follows the dependency graph, not wall-clock completion time
+
+**Knowing when to spawn a new main agent:**
+- When a slice is too large or discovers unexpected scope, it requests a breakdown from the coordinator
+- When a review finds a Critical finding, the coordinator spawns a dedicated fix agent with the finding + relevant context
+- When integration reveals a regression, coordinator spawns an investigation agent before retrying the merge
+
+**The coordinator's job (what stays in scripts, not LLM):**
+- Maintain the manifest (JSON file, append-only)
+- Compute the dependency graph
+- Decide parallelism vs serialization
+- Route: clean → merge, minor findings → fix agent, critical → escalate
+- Track worktrees, detect conflicts
+- Sequence the merge order
+
+**What requires LLM cognition:**
+- Discovery (what are the invariants, which files matter)
+- Design (which approach, what tradeoffs)
+- Implementation (write the code)
+- Review (is this correct and complete)
+- Breakdown (what are the right slice boundaries)
+
+**The minimum viable version:**
+A coordinator that handles a Medium/Small scoped task (already classified, no need for ideation or design). Takes 2-4 parallel slices, runs them, reviews each, merges when clean. No escalation handling in v1 -- if anything fails, notify the user.
+
+This is the thing that makes WorkTrain feel like a senior engineer taking ownership of a task, not a tool you have to supervise step by step.
+
+---
+
+### Coordinator design decision: MVP-first, generalize after (Apr 18, 2026)
+
+**Decision:** Build the first coordinator as a PR review-specific script. Generalize to a reusable coordinator framework after proving it works end-to-end.
+
+**Rationale:** Three discovery runs all converged on the architecture (TypeScript script, `CoordinatorDeps` interface, 2-call HTTP for notes). The risk is over-engineering for hypothetical pipelines before validating the real one. PR review is the highest-value first use case with a clear success criterion.
+
+**The generic coordinator architecture is already designed** (see `docs/discovery/coordinator-script-design.md`). The `CoordinatorDeps` interface and `AgentResult` bridge type make migration to a generic coordinator trivial -- the PR review script uses these types, so generalizing is additive, not a rewrite.
+
+**Migration path:** once PR review coordinator is proven in production, extract the routing logic (`parseFindings`, `routeByFindings`) and `CoordinatorDeps` interface into `src/coordinators/base.ts`. The PR review coordinator becomes one implementation of the base pattern.
+
+---
+
+### Architecture decisions from Apr 17-18 sessions (to record before files are cleaned up)
+
+**Decision 1: Structured output + tool calls can coexist (Apr 18)**
+Validated empirically via integration test. The beta API (`client.beta.messages.create()`) supports both JSON schema enforcement AND tool calls in the same request. Schema enforcement applies at `end_turn` only. Bedrock is more consistent than direct Anthropic API for system-prompt fallback behavior. This opens a future path for replacing `complete_step` with structured output, but `complete_step` remains the chosen primitive for now.
+
+**Decision 2: `complete_step` is the preferred daemon workflow-control primitive (Apr 18)**
+PR #569 merged. The daemon holds the continueToken in a closure; LLM calls `complete_step(notes)` and never handles the token directly. Structured output (`beta.messages.create` with JSON schema) was evaluated as an alternative and deferred -- it's a viable migration path for a future version but adds API complexity today. Follow-up: track a structured output migration as a future improvement, not a current priority.
+
+**Decision 3: AgentLoop error handling contract -- FatalToolError (Apr 16)**
+`FatalToolError` subclass selected for distinguishing recoverable from non-recoverable tool failures in the AgentLoop. The contract: user-facing tools (Bash, Read, Write) catch failures and return `isError: true` in the tool_result (loop continues, LLM can retry). Coordination tools with unrecoverable failures (session store corruption, token decode failure) throw `FatalToolError` -- `_executeTools` instanceof-checks this and kills the session rather than surfacing a confusing error to the LLM. This contract is part of the AgentLoop architecture and must be followed by any new tool implementations.
+
+**Decision 4: Use `wr.discovery` for discovery-only tasks, not `coding-task-workflow-agentic` (Apr 17)**
+Discovered from a broken session: `coding-task-workflow-agentic` dispatched with "do discovery only, no code" ran 11 step advances then stopped without `run_completed`. The workflow's implementation phases fired even with explicit instructions not to code. Lesson: when a trigger or coordinator wants pure discovery/research, use `wr.discovery` as the workflowId. `coding-task-workflow-agentic` should only be dispatched when implementation is the actual goal.
+
+**Decision 5: Bug -- MCP server EPIPE crash (Apr 18)**
+Root cause confirmed with 15 production crash log entries: `process.stderr` is missing an `'error'` event handler in `registerFatalHandlers()`. When an MCP client disconnects, Node.js emits `EPIPE` on stderr which crashes the process with an unhandled error. `process.stdout` already has equivalent protection via `wireStdoutShutdown()`. Fix: mirror the stdout protection for stderr. One-line fix being implemented in PR `fix/mcp-stderr-epipe-crash`.
+
+---
+
+### worktrain status → console integration (Apr 18, 2026)
+
+The `worktrain status` CLI command is Phase 1. Phase 2: the same data and rendering lives inside the console as the default landing view when you open it -- not the sessions list, the overview. Same `StatusDataPacket` type, two surfaces. The console overview replaces the need to run a CLI command; it auto-refreshes and stays live.
+
+---
+
+### WorkTrain as a native macOS app (Apr 18, 2026)
+
+Long-term vision: WorkTrain becomes a full native Mac app -- not just a CLI + web console, but a proper macOS application with a menubar icon, system notifications, windows, and native UX.
+
+**What this unlocks:**
+- Always-on menubar presence showing daemon status at a glance
+- Native macOS notifications (already built via osascript -- the app version uses UserNotifications framework directly)
+- The `worktrain status` overview as a native window, not a browser tab
+- Message queue and inbox as a native interface (type a message from anywhere on your Mac, not just the terminal)
+- Background daemon management -- start/stop/restart from the menubar without terminal
+- Deep system integration: file system events, calendar, Contacts, native share sheet
+
+**Tech stack options:**
+- Swift/SwiftUI: full native, best macOS integration, steeper learning curve from TypeScript
+- Electron + existing console UI: fastest path, same TypeScript codebase, but heavy
+- Tauri: Rust core + existing web frontend, lighter than Electron, good macOS support
+- React Native macOS: reuses React knowledge, not quite native feel
 
-**2. Kill the agent cleanly on detected failure**
-- When internal tooling is detected broken, stop the agent immediately -- do NOT let it continue without enforcement
-- Retain the full conversation history and step notes up to the point of failure
-- Mark the session as `interrupted_tooling_failure` (distinct from `error` or `timeout`)
-- Write the failure event to the daemon event log with the exact cause
+**Recommended path:** Tauri wrapping the existing console UI. The console is already a React/Vite app. Tauri gives native menubar, notifications, and system APIs without rewriting the frontend. The WorkTrain daemon stays as a separate process managed by the app.
 
-**3. Self-diagnose**
-- Run a lightweight health check: can we reach the session store? Can we decode the continueToken? Is the WorkRail engine responding?
-- Identify the root cause: MCP disconnect? Session store corruption? Token decode failure? Port conflict?
-- Distinguish recoverable (restart and resume) from non-recoverable (session data corrupted, must restart from scratch)
+**This is a post-v1 platform decision** -- not a near-term priority, but worth designing toward. Don't make architectural decisions that would make the Tauri wrapper hard later.
 
-**4. Fix and reboot**
-- For recoverable failures: restart the WorkRail engine in-process, re-register tools, verify health before resuming
-- For MCP-mode failures: reconnect without killing the parent session
-- For port conflicts: clear the lock and rebind
-- All of this happens automatically, without user intervention
+---
+
+### Long-running sessions: stay open across agent handoffs (Apr 18, 2026)
+
+**The problem:** today when an MR review session completes, it writes its findings and exits. If the findings require fixes, a new fix agent starts from scratch with no shared context. When the fix is done, a new re-review agent also starts from scratch. Three sessions that are logically one unit of work are isolated from each other.
+
+**The vision:** a session can stay open and wait -- dormant but alive -- while another agent does work. When that work completes, the waiting session resumes with full context continuity.
+
+**The MR review example:**
+
+```
+[MR review session]  finds: 2 critical, 3 minor
+  → stays open, waiting for fixes
+  
+  [Fix agent session]  addresses all 5 findings
+    → completes, signals "fixes ready"
+  
+[MR review session resumes]  re-reads the diff, re-evaluates
+  → all 5 verified fixed, 0 new findings
+  → completes with APPROVE verdict
+```
+
+The same session that found the issues verifies the fixes. No context reconstruction. No risk of re-review missing something the original reviewer knew.
+
+**Other use cases for waiting sessions:**
+
+- **Architecture review waiting for approval:** architect session identifies a design gap, waits for the human to decide on direction, resumes when the decision is recorded
+- **Discovery session waiting for data:** a research session identifies that it needs a specific file or API response, signals "blocked on: fetch X", waits for a retrieval agent to deliver it, resumes with the data injected
+- **Coordinator waiting on child completion:** instead of a coordinator script polling `worktrain await`, the coordinator session can yield and be resumed by the daemon when child sessions complete -- same session, same context, no polling overhead
+- **Spec authoring waiting for stakeholder input:** a spec session writes a draft, flags "needs: human review of acceptance criteria", waits, resumes when the human adds a comment
+- **Integration test waiting for deployment:** a test coordination session waits for a deploy to complete before running integration tests
+
+**The key insight: the LLM doesn't experience waiting.**
 
-**5. Resume with context**
-- Resume the session from the last confirmed `complete_step` / `advance_recorded` event
-- Inject a `<context>` block into the resumed session: "Your previous session was interrupted due to [reason]. Here is what you completed before the interruption: [last 3 step notes]. Resume from where you left off."
-- The agent never knows the interruption happened -- it just continues
-- If resume fails (token expired, session corrupted), escalate to the user with full context on what was completed and what wasn't
+LLMs have no concept of time. Between one turn and the next, zero time passes from the agent's perspective. This means "waiting" is not a thing that happens to the agent -- it just doesn't receive its next turn until the coordinator has something to give it.
 
-**6. Audit trail**
-- Every self-heal event is recorded in `~/.workrail/events/daemon/YYYY-MM-DD.jsonl` as `tooling_failure_detected`, `self_heal_started`, `self_heal_succeeded` / `self_heal_failed`
-- The console shows a `⚠️ self-healed` badge on sessions that were interrupted and resumed
-- The user can query: "which sessions were interrupted today?" and get the full list with causes
+The session is paused at the engine level (DAG holds at a node, no new turns issued). The agent submitted its output and simply hasn't received a response yet. When the coordinator is ready -- fix agent completed, human reviewed, deployment finished -- it advances the session with a turn that contains the new context. From the agent's perspective: it submitted findings and immediately received "here are the fixes, verify them."
+
+**No `wait_for` primitive needed at the workflow level.** The coordinator is the timing mechanism. This is the coordinator's job: know when each session is ready for its next input, and deliver that input at the right time.
+
+```
+Coordinator logic:
+
+1. Advance review session to "findings complete" node
+2. Read findings from session output
+3. Spawn fix agent with those findings
+4. Wait for fix agent to complete (worktrain await)
+5. Inject fix summary into review session's next turn
+6. Advance review session: "Here are the fixes. Verify them."
+   → LLM receives this as the natural next step, no time gap perceived
+```
+
+**Why this is more powerful than re-running a fresh session:**
+
+- **Context continuity:** the reviewer remembers what it found, why it flagged it, what invariants it was checking. A fresh session has to re-discover all of that.
+- **Relational memory:** "does this fix address the root cause I identified, or just the symptom?" -- only the original session knows the root cause reasoning.
+- **Efficiency:** no redundant context gathering. The resumed session picks up exactly where it left off.
+- **The agent doesn't know it's coordinating:** from the agent's view, it's a continuous workflow. The coordinator manages the timing externally.
 
 **Implementation path:**
-- Phase 1: detection only -- log `tooling_failure_detected` when a session produces output without `run_completed`. Surface in console and status command.
-- Phase 2: kill on detection -- stop agents immediately when tooling failure detected. No more unverified output reaching main.
-- Phase 3: auto-resume -- restart and resume for recoverable failures.
-- Phase 4: full self-heal loop -- diagnose, fix, reboot, resume automatically.
 
-**The invariant that must hold:** no output from a WorkTrain agent should be acted on (committed, merged, posted) unless its session has a confirmed `run_completed` event. This is the enforcement guarantee. Self-healing is what makes that guarantee survivable.
+- Phase 1: coordinator scripts withhold `complete_step` advancement until the condition is met. This already works today -- the coordinator just doesn't advance the session until the fix agent is done.
+- Phase 2: the coordinator passes structured context when advancing: `complete_step(session, { injectedContext: fixSummary })`. The session receives it as part of the next step's prompt.
+- Phase 3: declarative pipelines -- workflow JSON declares that step N waits for an external condition before proceeding. The coordinator reads this and manages the timing automatically. No hand-coded coordinator script needed for common patterns.
 
 ---
 
-### CRITICAL architectural clarity: three systems, one shared engine (permanent reference)
+### Coordinatable workflow steps: confirmation points the coordinator can satisfy (needs discovery, Apr 18, 2026)
+
+⚠️ **Needs discovery before implementation. The questions below are open, not answered.**
+
+**The insight:** workflows already have `requireConfirmation: true` on certain steps -- these are natural coordination points. Right now they pause for a human. The idea is to make them also pausable-for-a-coordinator, so a coordinator (or another agent) can be the one that responds instead of a human.
+
+**The vision:**
+A workflow reaches a `requireConfirmation` step. In MCP mode (human-driven), it behaves exactly as today -- pauses and waits. In daemon/coordinator mode, instead of blocking forever, the coordinator can:
+- Inject a synthesized answer based on external work it just did ("architecture review found X, proceed with approach A")
+- Spawn another agent to generate the answer and inject its output
+- Ask a discovery agent to weigh in and forward the result
+- Simply forward a human's message from the message queue
+
+The original session never knows whether a human or a coordinator satisfied the confirmation. It just receives the next turn with context.
+
+**Why this is powerful:**
+Today the coordinator is external to the workflow -- it orchestrates sessions from outside. This makes the workflow itself coordinatable from within, so multi-agent collaboration can be declared in the workflow spec rather than bolted on in coordinator scripts.
 
-This is the most important architectural fact about this codebase. Every agent and contributor must understand this before touching anything.
+**What's unknown and needs discovery:**
+1. **Mechanism:** is this an enriched `requireConfirmation` (add a `coordinatable: true` flag?), a new step type (`requireCoordinatorInput`?), or something at the engine level? Tradeoffs between each.
+2. **What gets injected:** always a structured decision ("proceed/revise/abort + findings"), or also data injection ("here are the file contents", "here's what the API returned")? How does the step receive it -- as a new tool call result, as a steer, as part of the step prompt?
+3. **Coordinator discovery:** how does the coordinator know a step is waiting for it vs waiting for a human? Does it poll the session state? Does the session emit a `coordinator_gate_pending` event? (This connects to the `waitForCoordinator` spec in this backlog.)
+4. **Timeout/fallback:** if the coordinator never responds, what happens? Fall back to human? Error? Configurable?
+5. **MCP invariant:** must behave identically to today in MCP/human-driven mode. The coordinator path is additive, not a behavior change for existing users.
 
-**WorkRail/WorkTrain is three separate systems sharing one engine:**
+**Relationship to other specs:**
+- "Long-running sessions: stay open across agent handoffs" -- the session pauses at the confirmation point, coordinator acts, session resumes
+- "POST /api/v2/sessions/:id/steer" -- this might be the injection mechanism
+- `signal_coordinator` tool -- the session might signal the coordinator instead of blocking
+- `waitForCoordinator` step flag (already in this backlog) -- same underlying need, different framing
+- "Coordinator review mode: self-healing vs comment-and-wait" -- confirmation points are where that routing decision gets expressed
 
+---
+
+## Architecture Decision: Three-Workflow Pipeline (Apr 18, 2026)
+
+### Decision
+
+The canonical WorkRail workflow pipeline for new features is:
+
+```
+wr.discovery (optional) → wr.shaping (optional) → coding-task-workflow-agentic
 ```
-                    Shared core
-          ┌─────────────────────────────┐
-          │  WorkRail engine            │
-          │  src/v2/durable-core/       │
-          │  ~/.workrail/data/sessions/ │
-          │  workflow registry          │
-          └────────┬──────────┬─────────┘
-                   │          │          │
-    ┌──────────────▼─┐  ┌─────▼──────┐  ┌▼─────────────┐
-    │ WorkRail MCP   │  │ WorkTrain  │  │ WorkRail     │
-    │ Server         │  │ Daemon     │  │ Console      │
-    │ workrail start │  │ worktrain  │  │ worktrain    │
-    │ src/mcp/       │  │ daemon     │  │ console      │
-    │                │  │ src/daemon/│  │ src/console/ │
-    │ Claude Code    │  │ src/trigger│  │              │
-    │ connects here  │  │            │  │ Shows BOTH   │
-    │ via stdio      │  │ autonomous │  │ MCP + daemon │
-    │                │  │ agent loop │  │ sessions     │
-    └────────────────┘  └────────────┘  └──────────────┘
+
+Each workflow is independently useful. The pipeline is an optional chain, not a required sequence.
+
+### Rationale
+
+**wr.discovery** produces a direction -- what problem is worth solving. Output: structured discovery notes at `.workrail/discovery/`.
+
+**wr.shaping** produces a bounded pitch -- what specifically to build and explicitly NOT build, at a product level. Output: `.workrail/current-pitch.md`. Faithful Shape Up methodology. Tech-agnostic. No code-level content.
+
+**coding-task-workflow-agentic** produces running code -- engineering approach, sliced implementation, verification. When pitch.md exists (Phase 0.5), it skips design ideation and translates the pitch directly into an engineering approach. The pitch's no-gos and appetite are binding constraints.
+
+### No TechSpec workflow needed
+
+The coding workflow already does everything a TechSpec workflow would do: Phase 1b generates design candidates, Phase 1c selects and challenges the approach, Phase 3 writes the spec and implementation plan. Adding a separate TechSpec workflow would duplicate this and create a question of which is canonical. The coding workflow is the engineering planning layer.
+
+**The split that matters is product vs engineering:**
+- Product decisions (what to build, for whom, within what time) → wr.shaping
+- Engineering decisions (how to build it, which interfaces, which tests) → coding workflow
+
+### When to skip shaping
+
+- Task is small, concrete, and clearly scoped → go straight to coding workflow
+- Discovery already produced a bounded, implementable direction
+- You have a pre-written ticket or spec that already defines what to build
+
+### Faithful Shape Up constraint
+
+wr.shaping is tech-agnostic. A pitch for a Kotlin Android app and a pitch for a Python API service look structurally identical. No file paths, no function signatures, no implementation details. This makes pitches usable by human engineering teams at companies using Shape Up, not just WorkRail's coding workflow.
+
+### Phase 0.5 mechanics
+
+When `coding-task-workflow-agentic` finds `.workrail/current-pitch.md`:
+1. Reads all five pitch sections (Problem, Appetite, Solution/Elements, Rabbit Holes, No-Gos)
+2. Sets `shapedInputDetected=true`
+3. Skips phases 1a-1c (hypothesis, design generation, challenge-and-select)
+4. Phase 1d translates pitch elements/invariants/no-gos into an engineering approach
+5. Plan audit (Phase 4) checks for drift against the pitch
+6. Appetite is a hard ceiling -- oversized engineering work becomes follow-up tickets
+
+
+---
+
+## Idea: `context-gather` Step Type (Apr 19, 2026)
+
+### Problem
+
+Phase 0.5 in the coding workflow currently looks for a shaped pitch by checking a local path. This doesn't handle: coordinator-injected context, manually written docs (GDoc, Confluence, Notion), Glean-indexed artifacts, or URLs embedded in the task description. The search logic is duplicated if other workflows need the same document.
+
+### Proposed primitive
+
+A new engine-level step type `context-gather` that resolves a named context artifact from ordered sources:
+
+```json
+{
+  "type": "context-gather",
+  "id": "gather-pitch",
+  "contextType": "shaped-pitch",
+  "outputVar": "shapedInput",
+  "optional": true,
+  "sources": ["coordinator-injected", "local-paths", "task-url", "glean"]
+}
 ```
 
-**WorkRail MCP Server:** `workrail start` → stdio MCP server. Claude Code connects here. Provides `start_workflow`, `complete_step`, `list_workflows` etc. as MCP tools. Source: `src/mcp/`. Must be bulletproof -- a crash kills all Claude Code workflow tools.
+**Source resolution order (stops at first hit):**
+1. `coordinator-injected` -- coordinator already attached context of this type to the session (most common in autonomous mode)
+2. `local-paths` -- check `.workrail/current-pitch.md`, `pitch.md`, `PRD.md`, `.workrail/pitches/` (most recent)
+3. `task-url` -- extract any URL from the task description and fetch via WebFetch or matching MCP (GDoc, Confluence, Notion)
+4. `glean` -- search Glean for recent docs matching the task keywords and `contextType`; opt-in only (risk of false positives silently constraining wrong scope)
+
+If `optional: true` and no source resolves: `outputVar = null`, workflow continues normally.
+
+### Why engine-level, not a routine
+
+- Coordinator intercept requires the engine to check "has this type already been provided?" before running any search -- a routine can't express that
+- `contextType` is a declared intent multiple workflows can share (`wr.shaping`, `coding-task-workflow`, `wr.discovery`) without duplicating resolver logic
+- New sources (Linear, Jira, Notion) get added to the engine once, immediately available to all workflows
+
+### Relationship to existing work
+
+- Replaces/supersedes Phase 0.5's current local-path check in `coding-task-workflow-agentic`
+- Coordinator PR-review flow would inject `shaped-pitch` context before spawning the coding session
+- Any workflow that needs "find the spec/pitch/PRD for this task" uses the same step type
+
+### Open questions
+
+- How does the coordinator inject context into a session? Via a session variable set before `start_workflow`, or a new `inject_context` call?
+- How does `task-url` distinguish a GDoc URL from a Confluence URL from a Notion URL? MCP routing by domain?
+- What is the `contextType` vocabulary? Start with `shaped-pitch` -- what else? (`discovery-notes`, `design-spec`, `api-contract`?)
+- Glean false-positive risk: wrong document fed as shaped input silently constrains wrong scope. Needs confidence threshold or explicit user confirmation when Glean is the only hit.
+
+
+---
+
+## Completed (Apr 19, 2026)
 
-**WorkTrain Daemon:** `worktrain daemon` → autonomous agent runner. Drives sessions without human involvement. Calls the WorkRail engine **directly in-process** -- does NOT go through the MCP server. Source: `src/daemon/`, `src/trigger/`.
+### wr.shaping -- Faithful Shape Up shaping workflow
 
-**WorkRail Console:** `worktrain console` → unified read-only session viewer. Shows sessions from **both** the MCP server and the daemon (they share the same session store). Requires neither the MCP server nor the daemon to be running.
+Created `workflows/wr.shaping.json`. Faithful Shape Up methodology, tech-agnostic, produces `.workrail/current-pitch.md` only. Nine steps: ingest → frame gate → diverge (6 shapes, Verbalized Sampling) → converge → breadboard + elements → rabbit holes + no-gos → draft/critique loop → approval gate → write pitch.md. Two human gates with autonomous fallback. Appetite is calendar-time only (xs/s/m/l/xl). No code-level content -- a pitch for a Kotlin app and a pitch for a Python service look structurally identical.
 
-**Rules that follow from this:**
-- MCP server code must never import from `src/daemon/` or `src/trigger/`
-- Daemon code must never depend on the MCP server being alive
-- Console code reads the session store directly -- no IPC with MCP or daemon needed
-- These are separate processes. A crash in one does not affect the others.
+### coding-task-workflow-agentic -- Upstream context Phase 0.5
+
+Added Phase 0.5 "Locate Upstream Context" to `coding-task-workflow-agentic.json`. Format-agnostic: the agent uses whatever tools are available (repo search, WebFetch, Confluence/Notion/Glean MCPs, etc.) to find any upstream document -- pitch, PRD, BRD, RFC, design doc, user story, Jira epic, etc. Sets `upstreamSpecDetected` + `solutionFixed` flags. When `solutionFixed=true`, design ideation phases (1a-1c) are skipped and Phase 1d translates upstream constraints directly into an engineering approach. Plan audit (Phase 4) checks for drift against `upstreamBoundaries` whenever an upstream document was found.
+
+Also consolidated from three workflow variants to one canonical file.
+
+
+---
+
+## Current state update (Apr 19, 2026)
+
+**npm version: v3.40.0**
+
+### What shipped since v3.36.0 (Apr 18 -- Apr 19)
+
+- ✅ **`wr.shaping`** -- faithful Shape Up shaping workflow (9 steps, two human gates with autonomous fallback)
+- ✅ **`coding-task-workflow-agentic` Phase 0.5** -- upstream context detection; skips design phases when solution is pre-specified. Three-workflow pipeline: shaping → discovery → coding.
+- ✅ **Coding workflow consolidated** -- from three variants (lean, full, lean.v2) to one canonical file.
+- ✅ **HttpServer removed from MCP server** (#601) -- pure stdio. MCP server can no longer accidentally start an HTTP server.
+- ✅ **Late-bound goals** (#604) -- `goalTemplate: "{{$.goal}}"` defaults for webhook-driven sessions. Goals can come from the payload, not just the static trigger definition.
+- ✅ **Coordinator message queue drain** (#606) -- `pr-review` coordinator reads `~/.workrail/message-queue.jsonl` before each spawn cycle. `worktrain tell stop`, `skip-pr <n>`, `add-pr <n>` work.
+- ✅ **Notifications shipped** -- `NotificationService` implemented, wired into `TriggerRouter` via `trigger-listener.ts`. `WORKTRAIN_NOTIFY_MACOS=true` and `WORKTRAIN_NOTIFY_WEBHOOK=<url>` in `~/.workrail/config.json`.
+- ✅ **`worktrain run pr-review`** -- fully wired coordinator command. `spawnSession` → `awaitSessions` → `getAgentResult` (session-wide artifact aggregation) → `parseFindingsFromNotes` → route by severity.
+- ✅ **`wr.review_verdict` artifact path** -- end-to-end wired: `mr-review-workflow.agentic.v2.json` phase-6 emits it, `artifact-contract-validator.ts` validates it at `continue_workflow` time, coordinator reads it with keyword-scan fallback.
+- ✅ **`worktrain logs` / `worktrain health`** -- structured daemon log tailing and per-session health summary. `worktrain status <id>` deprecated in favor of `worktrain health <id>`.
+- ✅ **`signal_coordinator` tool** -- agent can emit structured mid-session signals (`progress`, `finding`, `data_needed`, `approval_needed`, `blocked`) without advancing the step.
+- ✅ **`ChildWorkflowRunResult` + `assertNever`** -- spawn_agent delivery_failed bug fixed. `delivery_failed` impossible state is compile-time excluded.
+- ✅ **`lastStepArtifacts` on `WorkflowRunSuccess`** -- `onComplete` callback forwards artifacts alongside notes. Coordinator can read typed artifacts from result without a separate HTTP call.
+- ✅ **`steerRegistry` + POST `/sessions/:id/steer`** -- coordinator injection endpoint wired in daemon console. Running sessions register a steer callback; coordinators can inject mid-session messages via HTTP.
+- ✅ **GitHub polling adapters** -- `github_issues_poll` and `github_prs_poll` providers fully implemented alongside existing `gitlab_poll`.
+- ✅ **Knowledge graph spike** -- `src/knowledge-graph/` module: DuckDB in-memory + ts-morph indexer + two validation queries. NOT yet wired to an MCP tool (ts-morph in devDependencies).
+- ✅ **`worktrain daemon --install`** -- launchd plist creation, load, verify. Daemon survives MCP server reconnects.
+- ✅ **Performance sweep** -- April 2026 sweep identified 10 highest-leverage fixes, filed as issues #248-257. Not yet merged.
+
+### Accurate limitations (as of v3.40.0)
+
+1. **Console session tree UI not built** -- `parentSessionId` is stored in the `session_created` event and in `WorkflowRunSuccess`. Console `RunLineageDag` shows the per-session step DAG only. Cross-session parent-child tree is data-only. PRs #607 (tree view) and #608 (steer endpoint) are OPEN.
+2. **Daemon tool set is minimal** -- agent has: `complete_step`, `continue_workflow` (deprecated), `Bash`, `Read`, `Write`, `report_issue`, `spawn_agent`, `signal_coordinator`. No `Glob`, `Grep`, or `Edit`. Read/Write are thin wrappers.
+3. **`worktrain tell` messages only drained by coordinator** -- `drainMessageQueue` is called by `runPrReviewCoordinator`, not by the daemon loop. A running autonomous session cannot receive mid-run injections from `worktrain tell`. The `steerRegistry` HTTP endpoint is the mid-session channel.
+4. **Knowledge graph not wired** -- module exists, ts-morph must move to dependencies before an MCP tool can be built.
+5. **`spawn_agent` return missing `artifacts`** -- returns `{ childSessionId, outcome, notes }` only. Typed artifacts from child session are not surfaced to the parent agent. `lastStepArtifacts` on `WorkflowRunSuccess` exists but spawn_agent doesn't return it.
+6. **`worktrain inbox --watch` stub** -- `--watch` flag prints "not yet implemented" and exits.
+7. **Artifact store not built** -- agents still dump markdown/files directly into the repo. `~/.workrail/artifacts/` directory structure not created.
+8. **Performance issues not fixed** -- issues #248-257 filed from April sweep. `continue_workflow` triggers 6+ event log scans, full session rebuild per `/api/v2/sessions` request, N+1 workflow fetches, no caching.
+9. **No auto-commit** -- agents can write code but do not commit, push, or open PRs autonomously.
+10. **Assessment gates not battle-tested** -- end-to-end flow with `outputContract: required: true` not validated in production use.
+
+### Open PRs to merge
+
+- **#607** `feat(console): add session tree view for coordinator sessions` -- cross-session parent-child hierarchy in console. Blocked on: `parentSessionId` data is in store but console routes need to surface it.
+- **#608** `feat(console): add POST /api/v2/sessions/:sessionId/steer for coordinator injection` -- NOTE: this endpoint is already implemented in `daemon-console.ts` via `steerRegistry`. PR #608 may be adding this to the MCP server console separately. Check before merging.
+- **#610** `feat(workflows): add wr.shaping` -- the shaping workflow. Ready to merge.
+- **#587** `fix(mcp): add assertNever exhaustiveness guard to TriggerRouter` -- likely already applied in codebase (ChildWorkflowRunResult assertNever is live). May be a duplicate or different scope. Check.
+
+### Next priorities (groomed Apr 19)
+
+1. **Merge #610 (wr.shaping)** -- ready. Workflow is implemented and in the branch.
+2. **Merge #587 (TriggerRouter assertNever)** -- quick fix, check if still relevant.
+3. **Review and merge #607 + #608** -- console tree view and steer endpoint. Verify #608 doesn't duplicate what's already live in daemon-console.ts.
+4. **Performance fixes** -- issues #248-257. Pick highest-leverage first: SessionIndex (#248) and console projection cache (#249) eliminate most of the repeated scans.
+5. **Daemon tool set: add Glob + Grep** -- agents routinely need to search files. `Read` + `Bash` grep is slow and lossy. Native `Glob` and `Grep` tools would make coding sessions more reliable.
+6. **`spawn_agent` artifacts gap** -- add `artifacts?: readonly unknown[]` to the return value. `lastStepArtifacts` is already on `WorkflowRunSuccess`; wiring it through is ~30 LOC.
+7. **Knowledge graph wiring** -- move `ts-morph` and `@duckdb/node-api` to dependencies, add `query_knowledge_graph` MCP tool.
+8. **Artifact store foundation** -- `~/.workrail/artifacts/` directory, write path in `complete_step`.
 
 ---
 
-### CRITICAL architectural clarity: three systems, one shared engine (permanent reference)
+### wr.shaping workflow: shape messy problems into implementation-ready specs (needs authoring, Apr 18, 2026)
+
+**Status:** Design complete. Ready to author as a WorkRail workflow JSON.
+
+**Design docs:**
+- `docs/design/shaping-workflow-discovery.md` -- WorkRail-internal discovery findings
+- `docs/design/shaping-workflow-external-research.md` -- External research synthesis (Shape Up, LLM failure modes, artifact schema)
+
+**The gap this fills:** WorkRail has `wr.discovery` (divergent) and `coding-task-workflow-agentic` (convergent). Shaping is the missing middle -- converting messy discovery output into a bounded, implementation-ready spec without mid-implementation rabbit holes.
 
-This is the most important architectural fact about this codebase. Every agent and contributor must understand this before touching anything.
+**The 11-step skeleton (see design doc for full detail):**
+1. ingest_and_extract -- extract problem frames, forces, open questions
+2. **frame_gate** -- MANDATORY HUMAN GATE: confirm problem + appetite
+3. diverge_solution_shapes -- 4 parallel rough shapes with varied framings
+4. converge_pick -- SEPARATE JUDGE (different model/prompt): pick best shape
+5. breadboard_and_elements -- fat-marker breadboard + Interface/Invariant/Exclusion classification
+6. rabbit_holes_nogos -- adversarial: risks, mitigations, no-gos, assumptions
+7. context_pack_build -- file globs, reuse_utilities, conventions, do-not-touch boundaries
+8. example_map_and_gherkin -- Given/When/Then acceptance criteria + verification commands
+9. draft_pitch -- self-refine ×2, SEPARATE CRITIC (obfuscated authorship)
+10. **approval_gate** -- MANDATORY HUMAN GATE: approve, edit, or restart
+11. finalize_and_handoff -- schema validation, emit shape.json + pitch.md
 
-**WorkRail/WorkTrain is three separate systems sharing one engine:**
+**The single most important design decision:** generator and critic run on structurally different prompts (ideally different model families). CoT and self-reflection alone do NOT mitigate anchoring or self-preference bias (Lou & Sun 2025; Panickssery et al. 2024).
+
+**Output artifact:** `shape.json` -- contains problem story, appetite (multi-dimensional: calendar + tokens + turns + files), breadboard, elements, context_pack (file boundaries + reuse_utilities), Gherkin acceptance criteria, rabbit holes, no-gos, decomposition with walking skeleton, assumptions_log, build_readiness_score.
+
+**Key insight for AI implementers:** LLMs need MORE explicit specs than humans on interfaces/invariants/file boundaries (no tacit knowledge, no scope-shame), but LESS explicit than junior humans on standard patterns. The dominant failure mode is confident architectural divergence -- working code that reinvents an existing utility. Context Pack (Step 7) directly prevents this.
+
+**Next action:** author `wr.shaping` as a WorkRail workflow JSON using workflow-for-workflows, then update `coding-task-workflow-agentic` Phase 0 to detect and consume `shape.json` when present.
+
+---
+
+## Coordinator architecture: separation of concerns (Apr 19, 2026)
+
+**Decision: defer knowledge graph implementation until the context assembly layer is designed.**
+
+### The god class problem
+
+`src/coordinators/pr-review.ts` is already ~500 LOC doing: session dispatch, result aggregation, finding classification, merge routing, message queue drain, and outbox writes. Adding knowledge graph queries, context bundle assembly, upstream doc fetching, and prior session lookups would make it a god class.
+
+"Coordinator" is not a class or a script -- it is a **layer** that orchestrates across multiple concerns. Those concerns need to be separated before we add more to them.
+
+### The right layering
 
 ```
-                    Shared core
-          ┌─────────────────────────────┐
-          │  WorkRail engine            │
-          │  src/v2/durable-core/       │
-          │  ~/.workrail/data/sessions/ │
-          │  workflow registry          │
-          └────────┬──────────┬─────────┘
-                   │          │          │
-    ┌──────────────▼─┐  ┌─────▼──────┐  ┌▼─────────────┐
-    │ WorkRail MCP   │  │ WorkTrain  │  │ WorkRail     │
-    │ Server         │  │ Daemon     │  │ Console      │
-    │ workrail start │  │ worktrain  │  │ worktrain    │
-    │ src/mcp/       │  │ daemon     │  │ console      │
-    │                │  │ src/daemon/│  │ src/console/ │
-    │ Claude Code    │  │ src/trigger│  │              │
-    │ connects here  │  │            │  │ Shows BOTH   │
-    │ via stdio      │  │ autonomous │  │ MCP + daemon │
-    │                │  │ agent loop │  │ sessions     │
-    └────────────────┘  └────────────┘  └──────────────┘
+Trigger layer         src/trigger/          receives events, validates, enqueues
+Dispatch layer        (TBD)                 decides which workflow + what goal
+Context assembly      (TBD)                 gathers and packages context before spawning
+Orchestration layer   src/coordinators/     spawns, awaits, routes, retries, escalates
+Delivery layer        src/trigger/delivery  posts results back to origin systems
 ```
 
-**WorkRail MCP Server:** `workrail start` → stdio MCP server. Claude Code connects here. Provides `start_workflow`, `complete_step`, `list_workflows` etc. as MCP tools. Source: `src/mcp/`. Must be bulletproof.
+**Context assembly** is the missing layer. Before dispatching a coding session, something needs to:
+- Run `buildIndex()` and query "what imports the file being changed"
+- Find the upstream pitch/PRD/BRD for the task
+- Pull relevant prior session notes
+- Package everything as a structured context bundle
+
+This is NOT the orchestration script's job. The orchestration script should call `assembleContext(task, workspace)` and receive a bundle -- it should not know how that bundle was gathered.
+
+### Why the knowledge graph belongs in context assembly, not in the daemon
+
+Two options were considered:
+- **Daemon tool** (`makeQueryKnowledgeGraphTool` in `workflow-runner.ts`) -- agent queries mid-session on demand
+- **Coordinator pre-fetch** -- coordinator runs queries before spawning, injects answers as context
+
+The coordinator pre-fetch is better for known patterns (e.g. "what imports the file being changed" before a coding task). The agent doesn't need to know the graph exists -- it just gets the relevant facts as context. This also avoids adding `ts-morph` + DuckDB to the production build.
+
+The daemon tool approach is only better for ad-hoc mid-session queries the agent discovers dynamically. That's a secondary use case for v1.
+
+### What to build before the knowledge graph
 
-**WorkTrain Daemon:** `worktrain daemon` → autonomous agent runner. Drives sessions without human involvement. Calls the WorkRail engine **directly in-process**. Does NOT go through the MCP server. Source: `src/daemon/`, `src/trigger/`.
+1. **Design the `ContextAssembler` abstraction** -- takes task description + workspace + trigger metadata, returns a structured context bundle. The knowledge graph is one of several sources (alongside upstream docs, prior session notes, repo state).
+2. **Refactor `pr-review.ts`** to use a `ContextAssembler` for the bits that fit there.
+3. **Then** implement knowledge graph as a `ContextAssembler` plugin -- not as a coordinator script addition and not as a daemon tool.
 
-**WorkRail Console:** `worktrain console` → unified read-only session viewer. Shows sessions from **both** the MCP server and the daemon (shared session store). Requires neither to be running.
+### Anti-pattern to avoid
 
-**Rules:**
-- MCP server code must never import from `src/daemon/` or `src/trigger/`
-- Daemon code must never depend on the MCP server being alive
-- Console reads the session store directly -- no IPC with either needed
-- These are separate processes. A crash in one does not affect the others.
+Adding knowledge graph calls directly into `pr-review.ts` or any other coordinator script. That immediately creates the god class we're trying to avoid and couples the orchestration layer to a specific context source.