@exaudeus/workrail 3.59.3 → 3.59.5

package/docs/design/dispatch-dedup-prealloc-bypass-design-review.md

@@ -0,0 +1,100 @@
+# Design Review: Bypass Dispatch Dedup for Pre-Allocated Sessions
+
+**Date:** 2026-04-19
+**Reviewer:** Claude (automated design review pass)
+**Selected approach:** Wrap dedup block in `if (workflowTrigger._preAllocatedStartResponse === undefined)`
+
+---
+
+## Tradeoff Review
+
+### Shared dedup map stays shared
+
+The `_recentAdaptiveDispatches` map remains shared across all dispatch paths. Pre-alloc calls
+bypass the check but do not update the map.
+
+**Assessment:** Sound. The map entry from `dispatchAdaptivePipeline()` correctly blocks duplicate
+top-level pipelines for 30s. Pre-alloc calls are child sessions -- they should not add new map
+entries. All cross-path scenarios analyzed; no violations found.
+
+**Condition for unacceptability:** If a legitimate retry of the same goal+workspace (without
+pre-alloc) must fire within 30s of the original dispatch. Unlikely in practice; TTL is 30s.
+
+### Comment is the only regression protection
+
+The guard `if (_preAllocatedStartResponse === undefined)` wrapping the dedup block has no
+compile-time enforcement beyond the unit test.
+
+**Assessment:** Acceptable. The unit test `dispatch() with _preAllocatedStartResponse bypasses
+dedup and calls runWorkflowFn` catches any regression. The JSDoc on the field and the guard
+comment provide documentation-level protection.
+
+---
+
+## Failure Mode Review
+
+| Mode | Handled? | Mitigation |
+|---|---|---|
+| Guard removed in refactor | Yes | Unit test catches it |
+| Falsy check instead of `!== undefined` | Non-issue | Type is always an object when present |
+| Semaphore deadlock | Not new risk | Pre-existing FIFO semaphore handles this |
+| Session completes before enqueue | Not real | executeStartWorkflow doesn't run agent loop |
+
+**Highest-risk:** Guard removed in refactor. Mitigated by unit test.
+
+---
+
+## Runner-Up / Simpler Alternative Review
+
+- **Runner-up (early-return guard A):** Structurally equivalent. Loses because B keeps a single
+  enqueue block, reducing duplication risk. No elements worth borrowing beyond the guard comment.
+- **Simpler alternative (extract `_enqueueDispatch`):** Would be cleaner but is out of scope for
+  this targeted fix. No correctness benefit.
+
+---
+
+## Philosophy Alignment
+
+All relevant CLAUDE.md principles are satisfied:
+- **Architectural fixes over patches** -- the guard models the root invariant
+- **Make illegal states unrepresentable** -- compile-time discriminator
+- **YAGNI with discipline** -- minimal change, no speculation
+- **Document why, not what** -- guard comment explains invariant
+
+Pre-existing tensions (mutable shared map) are not introduced by this fix.
+
+---
+
+## Findings
+
+**No RED findings.** No blocking issues detected.
+
+**ORANGE (advisory):**
+1. The guard comment must be explicit about WHY dedup is bypassed, not just THAT it is bypassed.
+   A vague comment like `// skip dedup for pre-alloc` is insufficient. The comment must state:
+   'executeStartWorkflow already created the session in the store; dropping this dispatch would
+   zombie it.'
+
+**YELLOW (notes):**
+1. The unit test for the bypass case should assert that `runWorkflowFn` is called exactly once
+   (not just called), to verify the session actually starts.
+2. Consider adding a log line in the bypass path: `console.log('[TriggerRouter] Pre-allocated session dispatched: workflowId=...')` for observability.
+
+---
+
+## Recommended Revisions
+
+1. Write the guard comment to match the invariant stated in the design doc:
+   ```typescript
+   // Pre-allocated session: executeStartWorkflow already created the session in the store.
+   // Deduplication must not apply here -- dropping this dispatch would zombie the session.
+   ```
+2. In the test: assert `calls` has exactly 1 entry (`toHaveLength(1)`) after the bypass dispatch.
+3. Optional: add a `console.log` in the bypass path for daemon observability.
+
+---
+
+## Residual Concerns
+
+None. The fix is sound, minimal, and directly models the documented invariant. All failure modes
+are either handled or pre-existing. The design is ready for implementation.

package/docs/design/dispatch-dedup-prealloc-bypass-implementation-plan.md

@@ -0,0 +1,218 @@
+# Implementation Plan: Bypass Dispatch Dedup for Pre-Allocated Sessions
+
+**Date:** 2026-04-19
+**Branch:** fix/dispatch-dedup-prealloc-bypass
+**Scope:** src/trigger/trigger-router.ts only (src/mcp/ excluded)
+
+---
+
+## 1. Problem Statement
+
+`TriggerRouter.dispatch()` has a 30-second deduplication guard that compares incoming
+`goal::workspacePath` against `_recentAdaptiveDispatches`. When `dispatchAdaptivePipeline()`
+runs, it writes this key. Milliseconds later, `spawnSession()` calls `dispatch()` with the same
+goal and workspace (plus `_preAllocatedStartResponse`). The dedup guard fires, `queue.enqueue()`
+is never called, and the session that was already written to the store by `executeStartWorkflow()`
+zombies permanently.
+
+---
+
+## 2. Acceptance Criteria
+
+1. `dispatch()` called with `_preAllocatedStartResponse` set bypasses the dedup check and calls
+   `runWorkflowFn` exactly once.
+2. `dispatch()` called WITHOUT `_preAllocatedStartResponse` still deduplicates correctly within 30s.
+3. `npm run build` exits clean (no TypeScript errors).
+4. `npx vitest run tests/unit/trigger-router.test.ts` -- all tests pass including two new tests.
+5. `npx vitest run` -- no regressions in any other test file.
+6. PR merged to main via `gh pr merge <N> --squash`.
+7. Daemon rebuilt and reinstalled (`npm run build && node dist/cli-worktrain.js daemon --install`).
+8. `node dist/cli-worktrain.js trigger poll self-improvement` starts a session and `session_started`
+   appears in the event log within 30s.
+
+---
+
+## 3. Non-Goals
+
+- Do NOT touch `src/mcp/` (any file).
+- Do NOT implement Option B (remove dedup from dispatch() entirely).
+- Do NOT implement Option C (separate dedup maps).
+- Do NOT touch `route()` or `dispatchAdaptivePipeline()`.
+- Do NOT change the dedup TTL or map key format.
+
+---
+
+## 4. Philosophy-Driven Constraints
+
+- Guard comment must explain WHY, not what (CLAUDE.md: 'Document why, not what').
+- No code duplication: both the pre-alloc path and the normal path reach the same single
+  `queue.enqueue()` call (CLAUDE.md: 'Compose with small, pure functions').
+- Guard uses `!== undefined` not a falsy check (CLAUDE.md: 'Type safety as the first line of defense').
+
+---
+
+## 5. Invariants
+
+1. When `_preAllocatedStartResponse !== undefined`, `queue.enqueue()` MUST be called.
+2. When `_preAllocatedStartResponse === undefined`, the existing dedup check runs unchanged.
+3. `_recentAdaptiveDispatches` is not updated for pre-alloc dispatch calls (the entry from
+   `dispatchAdaptivePipeline()` remains and is the correct TTL anchor for top-level dedup).
+4. The `assertNever` exhaustiveness guard in the enqueue callback remains intact.
+
+---
+
+## 6. Selected Approach + Rationale
+
+**Approach:** Wrap the dedup block in `dispatch()` with:
+```typescript
+if (workflowTrigger._preAllocatedStartResponse === undefined) {
+  // ... existing dedup block ...
+}
+```
+Both the pre-alloc path and the normal (post-dedup) path fall through to the same single
+`void this.queue.enqueue(...)` call.
+
+**Rationale:** Minimal blast radius. Single guard. No code duplication. Directly models the
+documented invariant in `WorkflowTrigger._preAllocatedStartResponse` JSDoc.
+
+**Runner-up:** Early-return guard before the dedup block (Candidate A from design review).
+Lost because it risks duplicating the enqueue callback body. Structurally equivalent otherwise.
+
+---
+
+## 7. Vertical Slices
+
+### Slice 1: Implementation fix in trigger-router.ts
+
+**Scope:** `src/trigger/trigger-router.ts`, `dispatch()` method only (lines 847-933).
+
+**Change:**
+1. Add a guard comment before the dedup block explaining the pre-alloc invariant.
+2. Wrap the scoped dedup block `{...}` in `if (workflowTrigger._preAllocatedStartResponse === undefined)`.
+3. Add an optional `console.log` in the pre-alloc path for daemon observability.
+4. The existing `void this.queue.enqueue(...)` call remains after the if-block.
+
+**Acceptance criterion:** TypeScript compiles clean. The guard is visible and commented correctly.
+
+---
+
+### Slice 2: Unit tests in trigger-router.test.ts
+
+**Scope:** `tests/unit/trigger-router.test.ts`, new describe block or additions to existing
+'TriggerRouter.route and dispatch deduplication' describe block.
+
+**Two new tests:**
+
+**Test 1: dispatch() with _preAllocatedStartResponse bypasses dedup and calls runWorkflowFn**
+- Call `dispatchAdaptivePipeline(goal, workspace)` to prime the dedup map.
+- Then call `dispatch({ workflowId, goal, workspacePath: workspace, context: {}, _preAllocatedStartResponse: <fake> })`.
+- Flush the async queue.
+- Assert `calls.toHaveLength(1)` -- runWorkflowFn was called exactly once.
+
+**Test 2: dispatch() WITHOUT _preAllocatedStartResponse still deduplicates within 30s**
+- This test already exists at line 1604. Verify it still passes after the change.
+- No new test needed for this case; the existing test is the regression guard.
+
+**Acceptance criterion:** Both tests pass (`vitest run tests/unit/trigger-router.test.ts`).
+
+---
+
+### Slice 3: Build, test, PR, CI, merge
+
+**Steps:**
+1. `npm run build` -- clean
+2. `npx vitest run tests/unit/trigger-router.test.ts` -- all pass
+3. `npx vitest run` -- no regressions
+4. Create branch `fix/dispatch-dedup-prealloc-bypass`
+5. Commit: `fix(trigger): bypass dispatch dedup for pre-allocated sessions to prevent zombie sessions`
+6. Push + open PR
+7. Wait for CI
+8. Merge: `gh pr merge <N> --squash`
+
+---
+
+### Slice 4: Daemon reinstall and smoke test
+
+**Steps:**
+1. `npm run build && node dist/cli-worktrain.js daemon --install`
+2. `node dist/cli-worktrain.js trigger poll self-improvement`
+3. Watch for `session_started` in event log for at least 30s
+4. Confirm session is not zombie (status completes or progresses past `run_started`)
+
+---
+
+## 8. Test Design
+
+### New Test 1: Bypass case (primary regression test for this fix)
+
+```typescript
+it('dispatch(): bypasses dedup and calls runWorkflowFn when _preAllocatedStartResponse is set', async () => {
+  vi.useFakeTimers();
+  const { fn, calls } = makeFakeRunWorkflow();
+  const trigger = makeTrigger();
+  const router = new TriggerRouter(
+    makeIndex(trigger), FAKE_CTX, FAKE_API_KEY, fn,
+    undefined, undefined, undefined, undefined, undefined,
+    FAKE_DEPS, executors,
+  );
+
+  const goal = trigger.goal;
+  const workspace = trigger.workspacePath;
+
+  // Prime the dedup map via dispatchAdaptivePipeline
+  await router.dispatchAdaptivePipeline(goal, workspace);
+
+  // Now dispatch with _preAllocatedStartResponse set -- must bypass dedup
+  router.dispatch({
+    workflowId: trigger.workflowId,
+    goal,
+    workspacePath: workspace,
+    context: {},
+    _preAllocatedStartResponse: {} as any, // non-undefined value triggers bypass
+  });
+
+  // Flush
+  await new Promise((r) => setImmediate(r));
+
+  // runWorkflowFn must have been called exactly once
+  expect(calls).toHaveLength(1);
+  vi.useRealTimers();
+});
+```
+
+### Existing Test (regression guard): dispatch() dedup still works without _preAllocatedStartResponse
+
+The test at line 1604-1644 of trigger-router.test.ts covers this. It will serve as the
+regression guard after the change.
+
+---
+
+## 9. Risk Register
+
+| Risk | Likelihood | Impact | Mitigation |
+|---|---|---|---|
+| Guard removed in future refactor | Low | High | Unit test catches it in CI |
+| Comment becomes stale | Low | Medium | Comment explains invariant, not mechanics -- less likely to stale |
+| _preAllocatedStartResponse type changes | Very low | Low | TypeScript would catch it at compile time |
+
+---
+
+## 10. PR Packaging Strategy
+
+Single PR. One commit. No breaking changes.
+
+Branch: `fix/dispatch-dedup-prealloc-bypass`
+Commit: `fix(trigger): bypass dispatch dedup for pre-allocated sessions to prevent zombie sessions`
+
+---
+
+## 11. Philosophy Alignment
+
+| Principle | Status | Why |
+|---|---|---|
+| Architectural fixes over patches | Satisfied | Guard models the root invariant, not a special-case |
+| Make illegal states unrepresentable | Satisfied | `_preAllocatedStartResponse !== undefined` is compile-time discriminator |
+| YAGNI with discipline | Satisfied | Minimal change, no speculative abstractions |
+| Document why, not what | Satisfied | Guard comment explains invariant |
+| Type safety as first line of defense | Satisfied | `!== undefined` check, typed optional field |
+| Immutability by default | Tension (pre-existing) | Shared mutable map; not introduced by this fix |

package/docs/ideas/backlog.md

@@ -7299,3 +7299,55 @@ The daemon heartbeat + DaemonRegistry membership is the key insight: if the sess
 ### Priority
 
 Medium-high. True session status makes WorkTrain trustworthy as an autonomous system -- operators can see exactly what's happening without guessing. Especially important as session durations get longer (55-minute discovery sessions, 120-minute pipeline runs).
+
+---
+
+## Workflows tab: incorrect source attribution for bundled workflows (Apr 21, 2026)
+
+**The bug:** The Workflows tab in the console shows bundled workflows (e.g. `coding-task-workflow-agentic`, `workflow-for-workflows`) as coming from "User Library" instead of "WorkRail Built-in". This is a WorkRail MCP server issue, not a WorkTrain issue.
+
+**Likely cause:** The source attribution logic reads from the workflow's loaded source (the `WorkflowSource` type). When a workflow exists in both the bundled set AND a user's managed sources or remembered roots, the source returned is the one that "wins" in the storage layer -- which may be the user path rather than the bundled path. Or the `source.kind` field is incorrectly set to `'personal'` for workflows that were loaded from the bundled workflows directory.
+
+**Where to look:**
+- `src/infrastructure/storage/schema-validating-workflow-storage.ts` -- source kind propagation
+- `src/mcp/handlers/shared/workflow-source-visibility.ts` -- how source is mapped to display label in `list_workflows`
+- `src/infrastructure/storage/file-workflow-storage.ts` -- how `source.kind` is assigned when loading from disk
+
+**Expected behavior:** Workflows in the `workflows/` directory of the workrail package should always display as "WorkRail Built-in" regardless of whether the user also has a managed source that happens to include the same directory.
+
+**Priority:** Low for WorkTrain (doesn't affect functionality). Medium for WorkRail MCP (misleading UI, users may think they accidentally modified bundled workflows).
+
+---
+
+## Coordinator-managed git state and agent crash recovery (Apr 21, 2026)
+
+### Git state management (coordinator's job)
+
+Before dispatching any WorkTrain session that does git work:
+1. Check for `.git/index.lock` -- if present, verify the owning PID is dead (via `lsof` on macOS), then remove it
+2. Abort any in-progress git operations: `git rebase --abort; git merge --abort`
+3. Verify the workspace is in a clean state before handing off to the agent
+
+Every session that touches files gets a worktree (already implemented). The coordinator ensures worktrees are created cleanly and removed after session completion. The orphan TTL cleanup (24h) handles crash cases.
+
+### Agent crash recovery (coordinator's job)
+
+An agent can die from: stream watchdog timeout (600s no progress), OOM kill, or SIGKILL. In all cases the session event log is intact -- the full conversation history is preserved.
+
+**The coordinator should detect and recover automatically:**
+
+1. Monitor child sessions via `worktrain await`
+2. If a session returns `_tag: 'aborted'` or `_tag: 'timeout'` mid-pipeline:
+   - Check if the session made meaningful progress (step advances > 0, or notes written)
+   - If yes: resume the session -- same session ID, same context, agent picks up at last checkpoint
+   - If no (zero progress): retry from scratch with a fresh session, same context bundle
+3. Retry up to N times (configurable, default 2) before escalating to Human Outbox
+4. Track which phase failed and inject a hint on retry: "Previous attempt failed at this step. Retry with fresh approach."
+
+**This is session continuation applied to crash recovery.** The agent's conversation history is fully preserved. Resuming puts it back exactly where it was. The 600s watchdog timeout (the most common failure) almost always means a hung LLM call or a tool timeout -- resuming naturally retries the step.
+
+**Implementation:** `runFullPipeline` and `runImplementPipeline` already have per-phase error handling. Extend each `awaitSessions` call: on non-success outcome, attempt resume before returning escalated. The resume logic is `worktrain session continue <sessionId>` (once that command exists) or `dispatchAdaptivePipeline` with the existing session's context.
+
+### Priority
+
+High. Agent crash recovery makes the overnight-autonomous bar achievable. Without it, any hung LLM call or tool timeout fails the entire pipeline silently. With it, transient failures are automatically retried and the pipeline continues.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@exaudeus/workrail",
-  "version": "3.59.3",
+  "version": "3.59.5",
   "description": "Step-by-step workflow enforcement for AI agents via MCP",
   "license": "MIT",
   "repository": {