@exaudeus/workrail 3.27.0 → 3.29.0

package/docs/design/performance-sweep-2026-04.md

@@ -0,0 +1,96 @@
+# Performance Sweep -- April 2026
+
+**Date:** 2026-04-07
+**Status:** Discovery complete, issues filed
+
+Six parallel discovery agents audited the full workrail codebase for performance and efficiency issues. This document consolidates all findings.
+
+## Cross-cutting pattern
+
+Every layer independently re-reads and re-computes from raw data on every call. Nothing is shared between layers. The same session event log is scanned 10+ times per `continue_workflow` call across the engine, prompt renderer, and session store.
+
+## Findings by area
+
+### 1. Session store & persistence (`src/v2/infra/local/session-store/`)
+
+- `appendImpl` calls `loadTruthOrEmpty()` before every write -- a full manifest + all segment reads -- even though `ExecutionSessionGateV2` already loaded the session (double disk read per write)
+- Two separate `open/write/fsync/close` cycles when snapshot pins are present; should be one
+- 200 sequential `stat` calls in `readdirWithMtime` (for-loop, one at a time)
+- Segment files read sequentially despite being independent and immutable once written
+- `loadHealthySummaries` loads sessions sequentially with no concurrency cap and no cache
+- `validateAppendPlan` re-runs Zod parse on every event in the plan -- already trusted data
+- Full event payloads read in `loadTruthOrEmpty` just to extract `dedupeKey` fields
+- `new TextDecoder()` allocated per segment read (should be module-level singleton)
+- `mkdirp(eventsDir)` called on every `append`, not just session creation
+
+### 2. V2 engine core (`src/v2/durable-core/`, `src/mcp/handlers/v2-execution/`)
+
+- `continue_workflow` scans `truth.events` 6+ times per call across `continue-advance.ts`, `input-validation.ts`, `replay.ts` with no shared state
+- Session loaded from disk a second time after the advance completes; same events scanned again
+- `projectRunContextV2` called in `validateAdvanceInputs` then again inside `renderPendingPrompt`
+- `projectAssessmentsV2` runs full scan on every step even when no assessment events exist
+- Sortedness validation repeated in every projection (4+ times per advance) on data the store guarantees is sorted
+- `createWorkflow(pinned.definition)` called on every advance for the same immutable workflow hash -- never cached
+- `pinnedStore.get()` called twice on first-advance path when pin already found
+- `deriveWorkflowHashRef` called 3 times with the same input per advance
+- `hasPriorNotesInRun` adds a 4th+ event scan inside `renderPendingPrompt`
+
+### 3. Workflow loading & registry (`src/infrastructure/storage/`, `src/mcp/handlers/`)
+
+- N+1 `getWorkflowById` calls per `list_workflows`: 1 list + N individual fetches, then full 5-pass compilation + SHA-256 hash + disk read per workflow on every call
+- New AJV instance + schema compilation on every request (`createWorkflowReaderForRequest`)
+- Recursive filesystem walk of all remembered-root directories per request with workspace signal
+- `CachingWorkflowStorage` uses linear `find` scan instead of `Map` lookup
+- `listWorkflowSummaries` triggers full validation pass just to return metadata fields
+- `statSync` blocking event loop in index build (`FileWorkflowStorage.buildWorkflowIndex`)
+- `workflow.schema.json` re-read and JSON.parsed on every `workflow_get_schema` call
+- `listWorkflowSummaries` and `loadAllWorkflows` as two parallel independent index reads
+
+### 4. MCP handler layer (`src/mcp/handlers/`, `src/mcp/handler-factory.ts`)
+
+- Output schema `.parse()` on every hot-path response on data the server itself produced (handlers for `continue_workflow`, `start_workflow`, `list_workflows`, etc. all call `Schema.parse()` on their own output)
+- `V2BlockerReportSchema.superRefine()` runs O(n log n) duplicate-check on every parse
+- `process.env.WORKRAIL_CLEAN_RESPONSE_FORMAT` read as string comparison per call (not cached)
+- `coerceJsonStringObjectFields` rebuilds object-field set from schema shape per call
+- `JSON.stringify(..., null, 2)` with indentation on all machine-to-machine wire responses
+- `getV2ExecutionRenderEnvelope` called twice per non-execution response
+- Schema shape re-traversed on every validation error for suggestion generation
+
+### 5. Console service & data projection (`src/v2/usecases/console-service.ts`)
+
+- Full 500-session disk load + projection rebuild on every `/api/v2/sessions` request, no caching
+- `/api/v2/worktrees` calls `getSessionList()` a second time (double the I/O)
+- `projectRunDagV2` called 3-4 times on the same event array per session per request
+- `resolveRunCompletion` always re-projects the DAG from events even when caller has it
+- `projectRunStatusSignalsV2` internally calls `projectRunDagV2` + `projectGapsV2` again
+- `projectSessionHealthV2` calls `projectRunDagV2` yet again
+- `projectNodeOutputsV2` called twice per session summary (title extraction + recap)
+- `projectNodeDetail` runs 5 independent full event-log scans sequentially
+- `loadSegmentsRecursive` O(N^2) array allocations via spread per segment
+
+### 6. Prompt rendering & content assembly (`src/v2/durable-core/domain/`)
+
+- `renderPendingPrompt` runs 3 independent full-event-log projections (`projectRunContextV2`, `projectRunDagV2`, `projectNodeOutputsV2`) plus `hasPriorNotesInRun` scan
+- `resolveParentLoopStep` and `getStepById` both do double-nested workflow traversal on every render
+- `expandFunctionDefinitions` re-searches workflow definition on every call
+- `buildChain`/`buildPathBackward` in `recap-recovery.ts` allocate O(N^2) `Set` objects per ancestry traversal
+- `renderBudgetedRehydrateRecovery` encodes the same string 3 times in the budget-trim loop
+- Tier lookup functions use `Array.find` over constant 2-3 element arrays (should be `Record`)
+- Shared mutable global `g`-flag regex in `context-template-resolver.ts` (latent correctness bug)
+- `dotPath.split('.')` allocates new array on every template token match
+- `JSON.stringify` for node deduplication equality in `projectRunDagV2`
+
+## Highest-leverage fixes
+
+| Priority | Fix | Areas | Issues |
+|---|---|---|---|
+| 1 | `SessionIndex`: build once at load, thread through engine + renderer | Engine, renderer | #248 |
+| 2 | `(sessionId, mtime)` projection cache in console service | Console | #249 |
+| 3 | Remove output-side Zod `.parse()` on server-produced responses | MCP | #250 |
+| 4 | Thread loaded session into `appendImpl` (eliminate double disk read) | Session store | #252 |
+| 5 | Cache `createWorkflow` by hash; fix AJV singleton; fix fs walk | Engine, workflows | #254, #256 |
+| 6 | Parallelize serial I/O (stat loop, segment reads) | Session store | #253 |
+| 7 | Pre-index step/loop/function lookups at Workflow construction | Engine, renderer | #255 |
+| 8 | Fix N+1 workflow fetches and recursive fs walk per request | Workflows | #256 |
+| 9 | Fix serialization overhead (JSON indent, env vars, coercion) | MCP | #251 |
+| 10 | Fix O(N^2) ancestry + budget loop re-encoding + minor allocations | Renderer | #257 |

package/docs/design/routines-guide.md

@@ -0,0 +1,219 @@
+# Routines Guide — Three Consumption Modes
+
+Routines are reusable cognitive workflows defined as JSON in `workflows/routines/`.
+They can be consumed in three ways, each suited to different orchestration needs.
+
+## Mode 1: Delegation (WorkRail Executor)
+
+The primary agent delegates a routine to a **WorkRail Executor subagent** at runtime.
+The subagent runs the routine's steps independently and returns output to the parent.
+
+**When to use**: bounded cognitive tasks (design generation, hypothesis challenge, plan analysis)
+where the parent agent wants to continue working in parallel.
+
+**How it works**:
+1. Parent agent spawns a WorkRail Executor with a `routineId`
+2. The executor runs the routine's steps sequentially
+3. Output flows back to the parent via the session
+
+**Example** (in a workflow step prompt):
+```
+Spawn ONE WorkRail Executor running `routine-tension-driven-design` with your
+tensions, philosophy sources, and problem understanding as input.
+```
+
+## Mode 2: Direct Execution (Agent Follows Steps)
+
+An agent reads the routine definition and **follows its steps directly** as structured guidance.
+No subagent spawning — the agent itself executes each step in sequence.
+
+**When to use**: when the agent IS the executor (e.g., inside a WorkRail Executor session),
+or when delegation overhead isn't justified.
+
+**How it works**:
+1. Agent loads the routine JSON
+2. Agent executes each step's prompt in order
+3. Agent produces the deliverable described in the final step
+
+## Mode 3: Injection (Compile-Time Template Expansion)
+
+A workflow references a routine via a `type: "template_call"` step, and the **compiler expands the routine's
+steps inline** at compile time. The routine's steps become first-class workflow steps.
+
+**When to use**: when routine steps should be visible in the workflow's step list, participate
+in confirmation gates, and be tracked individually in the session.
+
+**How it works**:
+1. A workflow step declares a `type: "template_call"` step with the routine's template ID and args
+2. At compile time, the template registry expands the routine into real steps
+3. The expanded steps replace the template call step in the compiled workflow
+4. `{arg}` placeholders in prompts are substituted; `{{contextVar}}` is preserved for runtime
+
+**Template ID convention**:
+- Routine `routine-tension-driven-design` → template ID `wr.templates.routine.tension-driven-design`
+- The `routine-` prefix is stripped automatically
+
+**Example** (in workflow JSON):
+```json
+{
+  "type": "template_call",
+  "templateId": "wr.templates.routine.tension-driven-design",
+  "args": {
+    "deliverableName": "design-candidates.md"
+  }
+}
+```
+
+**What happens at compile time**:
+- The step above is replaced by the routine's 5 steps (step-discover-philosophy, step-understand-deeply, etc.)
+- Each expanded step ID is prefixed using the compiler's provenance/step identity rules
+- `{deliverableName}` in prompts becomes `design-candidates.md`
+- The routine's `metaGuidance` is injected as step-level `guidance` on each expanded step
+- `preconditions` and `clarificationPrompts` are NOT included (parent workflow handles those)
+
+**Constraints**:
+- Routine steps must NOT contain nested `template_call` usage (no recursive injection)
+- All `{arg}` placeholders must be satisfied by the template call's `args`
+- Arg values must be primitives (string, number, boolean) — objects/arrays are rejected
+
+## Comparison
+
+| Aspect | Delegation | Direct Execution | Injection |
+|---|---|---|---|
+| When resolved | Runtime | Runtime | Compile time |
+| Parallelism | Yes (subagent) | No | N/A (steps are inline) |
+| Step visibility | Opaque to parent | Transparent | Fully visible |
+| Confirmation gates | Subagent only | Agent decides | Per-step as authored |
+| Session tracking | Separate session | Same session | Same session, per-step |
+| Arg substitution | Via context | Via context | `{arg}` → compile-time |
+
+## Selection guidance
+
+Choosing the right consumption mode matters as much as choosing the right routine.
+
+### Default decision rule
+
+Use this order unless you have a strong reason not to:
+
+- **Use injection (`templateCall`) by default** when the routine is part of the parent workflow's authored structure.
+- **Use delegation** when the routine's value comes from an independent perspective, parallelism, or an intentionally opaque bounded audit.
+- **Use extension points only to make delegation seams overridable**, not as a substitute for routine injection.
+
+A good litmus test:
+
+- If you want the routine's **steps to appear in the parent workflow**, use **injection**.
+- If you want the routine's **result but not its internal steps**, use **delegation**.
+- If you want a team to **swap which delegated implementation is called** without forking the parent workflow, add an **extension point** around that delegated seam.
+
+### What extension points do not do
+
+`extensionPoints` and `{{wr.bindings.*}}` do **not** inject a routine into the parent workflow.
+
+They only resolve a slot to a routine/workflow ID in prompt text at compile time. The parent agent still decides whether to call or follow that bound implementation at runtime.
+
+Because binding resolution runs **after** template expansion, extension points cannot currently choose which routine gets injected via `templateCall`.
+
+### Prefer delegation when
+
+- an independent cognitive perspective adds value
+- the parent can continue useful work in parallel
+- the routine is acting as an auditor, challenger, or verifier
+- the routine's internal steps do not need to be visible as first-class parent workflow steps
+
+Common examples:
+
+- context completeness / depth audits
+- adversarial hypothesis challenge
+- philosophy alignment review
+- final verification from a fresh perspective
+
+### Prefer direct execution when
+
+- delegation overhead is not justified
+- the current agent is already the natural executor
+- step visibility is unnecessary
+- the routine is mainly a reusable thinking scaffold, not a separate perspective
+
+### Prefer injection when
+
+- the routine's steps should be visible in the parent workflow
+- confirmation behavior should apply per injected step
+- session traceability matters
+- the routine is central enough to the parent workflow that hiding it behind opaque delegation would reduce debuggability
+- the author wants the engine, not the agent, to own that reusable subflow
+
+Common examples:
+
+- reusable design-generation cores
+- reusable final-verification skeletons
+- bounded reusable subflows the author wants Studio/session visibility for
+
+## Auditor-first guidance
+
+For many high-value routines, the best default mental model is **auditor**, not **task owner**.
+
+That means the parent workflow:
+
+- gathers or synthesizes the current state
+- delegates a bounded audit/challenge/verification package
+- interprets the returned artifact as evidence
+
+not as canonical truth.
+
+This is often a better fit than executor-style delegation for:
+
+- review workflows
+- planning workflows
+- verification-heavy workflows
+
+## High-value routine defaults
+
+The current routine catalog suggests these default uses:
+
+- `routine-context-gathering`: completeness/depth audit or bounded context expansion
+- `routine-hypothesis-challenge`: adversarial challenge against the current leading story
+- `routine-execution-simulation`: bounded runtime/flow reasoning where mental execution adds value
+- `routine-philosophy-alignment`: review against user/repo principles
+- `routine-final-verification`: proof-oriented end-state validation
+
+## Good and bad fits
+
+### Good fit for delegation
+
+- an adversarial reviewer challenging the current recommendation
+- a philosophy/policy auditor checking alignment against repo rules
+- a fresh final verifier evaluating whether evidence really supports the conclusion
+
+### Bad fit for delegation
+
+- tiny deterministic transformations that the parent can do faster directly
+- parent-owned loop decisions or canonical synthesis
+- work where hiding the internal steps would make the session harder to debug
+
+### Good fit for injection
+
+- a reusable multi-step authoring scaffold the parent wants visible in the step list
+- a reusable verification sequence that should honor parent confirmation gates
+
+### Bad fit for injection
+
+- every small repeated instruction block
+- routines whose value comes mainly from independent perspective rather than visible sub-steps
+
+### Prefer extension points when
+
+- the parent workflow intentionally delegates a bounded seam
+- teams may want to replace that delegated implementation per project
+- the parent workflow still owns synthesis, loop control, and final decisions
+
+### Bad fit for extension points
+
+- using `{{wr.bindings.*}}` where the real goal is inline routine structure
+- hiding a core parent subflow behind a rebinding slot just to avoid hardcoding a routine ID
+- expecting project bindings to change which routine a `templateCall` injects
+
+## See Also
+
+- `workflows/examples/routine-injection-example.json` — example workflow using injection
+- `src/application/services/compiler/template-registry.ts` — injection implementation
+- `src/application/services/compiler/routine-loader.ts` — routine loading from disk

package/docs/design/sequence-diagrams.md

@@ -0,0 +1,11 @@
+# Sequence Diagrams for Native Context Management
+
+> **Not pursuing**
+>
+> WorkRail is not planning to implement native context management.
+>
+> This file is kept only as a stable tombstone so old links do not break.
+>
+> See:
+> - `docs/roadmap/legacy-planning-status.md`
+> - `docs/plans/native-context-management-epic.md`

package/docs/design/subagent-design-principles.md

@@ -0,0 +1,220 @@
+# Subagent Design Principles & Catalog
+
+## Overview
+
+This document defines WorkRail's approach to subagent design for agentic IDEs. It outlines the core principles, patterns, and catalog of specialized subagents that enhance WorkRail workflows.
+
+**Philosophy:** Subagents are **specialized cognitive functions**, not task owners. They execute complete, autonomous routines and return structured deliverables to the main agent orchestrator.
+
+---
+
+## Core Principles
+
+### 1. **Cognitive Specialization, Not Task Ownership**
+
+** Good:** "Context Researcher" - Specializes in deep reading and systematic exploration
+** Bad:** "Debugger" - Too broad, owns entire debugging workflow
+
+**Rule:** Subagents should embody a **specific cognitive mode** (exploration, challenge, verification) that can be applied across many workflows, not own a complete workflow themselves.
+
+### 2. **Stateless & Self-Contained**
+
+Each subagent invocation is independent:
+- **No memory** between calls
+- **No conversational refinement**
+- **No follow-up questions**
+
+**Implication:** The main agent must provide **all necessary context upfront** in a single, complete work package.
+
+**Pattern:**
+```
+Main Agent → Subagent: [Complete Context Package]
+Subagent: [Autonomous Execution]
+Subagent → Main Agent: [Structured Deliverable]
+```
+
+### 3. **Autonomous Routine Execution**
+
+Subagents execute **complete routines** from start to finish:
+- Receive: Self-contained work package with all context
+- Execute: Multi-step routine autonomously
+- Return: Named, structured artifact (e.g., `ExecutionFlow.md`)
+
+**Not this:** Iterative back-and-forth, gradual context building, conversational refinement.
+
+### 4. **Depth-Aware Investigation**
+
+For research/exploration tasks, subagents support **configurable depth levels** to balance speed vs thoroughness:
+
+| Level | Name | Time | Use Case |
+|-------|------|------|----------|
+| 0 | Survey | 1-2 min | "What exists here?" |
+| 1 | Scan | 5-10 min | "What are the major components?" |
+| 2 | Explore | 15-30 min | "What does each component do?" |
+| 3 | Analyze | 30-60 min | "How does this specific logic work?" |
+| 4 | Dissect | 60+ min | "What is every line doing?" |
+
+Main agent chooses depth based on uncertainty and importance.
+
+### 5. **Structured Deliverables**
+
+Every subagent routine produces a **named artifact** with a **consistent structure**:
+
+**Standard Output Format:**
+```markdown
+### Summary (3-5 bullets)
+- Key findings
+
+### Detailed Findings
+- Component breakdowns
+- File citations (file:line)
+
+### Suspicious Points / Concerns / Gaps
+- What could be problematic
+- What couldn't be determined
+
+### Recommendations
+- What main agent should do next
+```
+
+**Deliverable Quality Gates:**
+
+Main agent validates each deliverable against these criteria:
+-  **Completeness**: All required sections present
+-  **Citations**: File:line references for all findings
+-  **Gaps Section**: Explicit about limitations and unknowns
+-  **Actionability**: Clear next steps or recommendations
+
+**If a deliverable fails quality gates**, the main agent should:
+1. Note the gaps in the workflow context
+2. Decide if the partial deliverable is sufficient
+3. Optionally re-run with clarified context (not automatic)
+
+**Artifact Naming Convention:** Use kebab-case for filenames:
+- `execution-flow.md`
+- `hypothesis-challenges.md`
+- `plan-analysis.md`
+
+### 6. **Explicit Over Implicit**
+
+While agentic IDEs support auto-invocation (system picks subagent based on task description), **WorkRail workflows use explicit delegation**:
+
+```
+Use: task(subagent_type="context-researcher", prompt="...")
+Not: "Hey, someone gather context for me" (auto-invoke)
+```
+
+**Rationale:** Predictability, debuggability, user understanding.
+
+### 7. **Auditor Model: Review, Don't Execute**
+
+**Key Discovery:** Subagents work better as **auditors** than **executors**.
+
+** Executor Model (Problematic):**
+```
+Main Agent: "Go gather context about authentication"
+Subagent: *reads files, builds understanding*
+Problem: Main agent doesn't have the context, needs to re-read
+```
+
+** Auditor Model (Effective):**
+```
+Main Agent: *reads files, builds understanding*
+Main Agent: "I read these files and learned X. Audit my work."
+Subagent: "You missed Y, assumption Z is risky, go deeper on W"
+Main Agent: *investigates gaps*
+Result: Main agent has full context + quality control
+```
+
+**Why Auditors Work Better:**
+- **No dilution**: Main agent has full, uncompressed context
+- **No duplication**: Main agent doesn't need to re-read what subagent read
+- **Fresh perspective**: Auditor catches gaps and blind spots
+- **Quality control**: Ensures sufficient understanding before proceeding
+- **Cognitive diversity**: Different perspective on the same work
+
+**When to Use Auditors:**
+- Context gathering (audit for completeness and depth)
+- Hypothesis formation (challenge assumptions)
+- Plan creation (validate completeness and soundness)
+- Final validation (adversarial review before committing)
+
+**When Executors Still Make Sense:**
+- Simulation (running "what-if" scenarios in parallel)
+- Independent parallel work (different execution paths)
+- Specialized tasks main agent can't do well
+
+### 8. **Parallel Delegation for Critical Work**
+
+**Pattern:** Spawn multiple subagents **simultaneously** for critical phases to get diverse perspectives and ensure nothing is missed.
+
+**Explicit Parallelism:**
+```
+ **CRITICAL: Spawn ALL subagents SIMULTANEOUSLY, not sequentially.**
+
+Delegate to THREE subagents AT THE SAME TIME:
+1. [Subagent 1 with specific focus]
+2. [Subagent 2 with different focus]
+3. [Subagent 3 with different focus]
+```
+
+**Use Cases:**
+
+**1. Multi-Perspective Auditing (Diverse Focuses)**
+```
+Main agent gathers context
+↓
+Parallel Audit (2-3 subagents):
+├─ Context Researcher (FOCUS: Completeness)
+├─ Context Researcher (FOCUS: Depth)
+└─ [Optional 3rd perspective]
+
+Main agent synthesizes all perspectives
+```
+
+**2. Redundant Critical Work (Different Rigor)**
+```
+Main agent forms hypotheses
+↓
+Parallel Challenge (2 subagents):
+├─ Hypothesis Challenger (rigor=3: Thorough)
+└─ Hypothesis Challenger (rigor=5: Maximum)
+
+Main agent strengthens hypotheses based on challenges
+```
+
+**3. Multi-Modal Validation (Different Cognitive Modes)**
+```
+Main agent proposes fix
+↓
+Parallel Validation (3 subagents):
+├─ Hypothesis Challenger (adversarial review)
+├─ Execution Simulator (simulate the fix)
+└─ Plan Analyzer (validate the plan)
+
+Main agent proceeds only if ALL THREE validate
+```
+
+**Synthesis Guidance:**
+
+When main agent receives multiple parallel deliverables:
+- **Common concerns**: If 2+ subagents flag the same issue → High priority
+- **Unique insights**: Each subagent may catch different gaps → Investigate all
+- **Conflicting advice**: If they disagree → Investigate to understand why
+- **Quality gate**: For critical phases, require ALL subagents to validate
+
+**Cost/Speed Tradeoff:**
+- Parallel = faster wall time but higher token cost
+- Use for critical phases where quality matters most
+- Use for phases where diverse perspectives add value
+
+### 9. **Focused Audits for Parallel Work**
+
+When spawning multiple auditors in parallel, give each a **specific focus** to maximize diversity and minimize overlap.
+
+**Pattern:**
+```
+Subagent 1: FOCUS = Completeness
+- Priority: Did they miss any critical areas?
+- Still checks other dimensions, but emphasizes coverage
+```