valent-pipeline 0.2.18 → 0.2.20

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "valent-pipeline",
-  "version": "0.2.18",
+  "version": "0.2.20",
   "description": "v3 multi-agent AI pipeline for software development lifecycle",
   "type": "module",
   "bin": {

package/pipeline/docs/lead-lifecycle.md

@@ -69,10 +69,15 @@ Spawn teammates from `agents-manifest.yaml` with:
 
 The lead watches the board, not the work. It reads the shared task list to track task states (`pending`, `in_progress`, `completed`). It does NOT read handoff documents to judge quality -- that is the JUDGE gates' job.
 
+### Heartbeat Timer
+
+At the start of Phase 2, the lead creates a `CronCreate` job that fires every 4 minutes (aligned within Claude's 5-minute prompt cache TTL for near-zero token cost). Each heartbeat triggers a liveness check: call `TaskList`, verify at least one non-Knowledge agent is `in_progress`. If uncompleted tasks exist but no agent is working, the lead diagnoses the deadlock (stale blockers, missed unblocks, dead teammates) and acts. A separate 4-minute keep-alive ping is created for the Knowledge agent to prevent its prompt cache from expiring during idle periods. Both cron jobs are deleted during Phase 3 teardown.
+
 ### What the Lead Watches
 
 - Task status transitions on the shared task list
 - Time-in-state for stall detection
+- Heartbeat liveness checks (every 4 minutes via cron timer)
 - Inbox messages from teammates (escalations, questions, completion notifications)
 
 ### What the Lead Does NOT Do

package/pipeline/prompts/knowledge.md

@@ -50,6 +50,8 @@ Verify ChromaDB at `{chromadb_host}` using v2 API. Base path: `{chromadb_host}/a
 ## Query Handling
 
 For each incoming query:
+
+0. If query is `cache-keepalive`: respond `[KNOWLEDGE-RESPONSE] ack` and stop. This is a prompt cache keep-alive ping — do not search.
 1. Search correction directives for relevant entries
 2. Search curated knowledge files for matching sections
 3. If database connected (SQLite mode): query using the CLI tool and read stdout for results:

package/pipeline/prompts/lead.md

@@ -49,6 +49,7 @@ You receive many message types. Process each by type:
 | `[DESIGN-COUNCIL]` | Agent requests deliberation. Route to relevant peers or participate. |
 | `[ESCALATION]` | Agent requests your intervention. Assess and act. |
 | `[BUG]` | QA-B filed a bug. No action -- QA-B routes directly to devs. Monitor bug count. |
+| `[HEARTBEAT]` | Timer-fired liveness check. Run the Heartbeat Liveness Check procedure below. |
 
 Your outbound messages follow the same terse format:
 - `[SPAWN] Spawning {agent} for {story_id}. Role: {role}. Shared context: {story_output_dir}.`
@@ -431,6 +432,50 @@ Wait for `[KNOWLEDGE-READY]` response before spawning other agents. The Knowledg
 
 You watch task status, NOT agent outputs. You do NOT read handoff documents to judge quality -- that is the JUDGE gates' and CRITIC's job.
 
+### Heartbeat Setup
+
+At the start of Phase 2, create a recurring heartbeat timer using `CronCreate`:
+
+```
+CronCreate:
+  cron: "*/4 * * * *"
+  prompt: "[HEARTBEAT] Run liveness check — verify at least one agent is actively working."
+  recurring: true
+```
+
+This fires every 4 minutes — aligned to stay within Claude's 5-minute prompt cache TTL so each heartbeat is near-zero cost. Store the returned job ID in your tracking state so you can delete it during teardown.
+
+### Knowledge Cache Keep-Alive
+
+Knowledge is a long-lived reactive agent that can sit idle for extended periods between queries. To prevent its prompt cache from expiring (5-minute TTL), create a separate recurring ping:
+
+```
+CronCreate:
+  cron: "*/4 * * * *"
+  prompt: "[KNOWLEDGE-QUERY] cache-keepalive"
+  recurring: true
+```
+
+Send this to Knowledge's inbox (not to Lead). Knowledge will respond with a no-op `[KNOWLEDGE-RESPONSE]` — the round-trip keeps the cache warm. Store the job ID alongside the heartbeat job ID and delete both during teardown.
+
+### Heartbeat Liveness Check
+
+When you receive a `[HEARTBEAT]` message:
+
+1. Call `TaskList` to get current task states
+2. Count tasks that are `in_progress` (exclude Knowledge Agent — it is reactive and has no task)
+3. Count tasks that are NOT `completed` (pending or in_progress)
+4. Evaluate:
+   - **All tasks completed:** All work is done. Proceed to Phase 3 if JUDGE has approved. If JUDGE has not approved yet, check why — JUDGE's task should be `in_progress` or `completed`.
+   - **Uncompleted tasks exist AND at least one is `in_progress`:** Healthy. No action needed.
+   - **Uncompleted tasks exist AND zero are `in_progress`:** All agents are idle with work remaining. This is the deadlock edge case. Diagnose:
+     a. Check which tasks are `pending` and what they are `blockedBy`
+     b. Verify the blocking tasks are truly not completed (task state may be stale)
+     c. If a blocker is completed but the downstream task was not unblocked, unblock it now
+     d. If an agent should be working but is not, send `[CHECK-IN]` to that teammate
+     e. If a teammate has died (no response to `[CHECK-IN]`), respawn it using crash recovery (see Phase 4)
+     f. If the dependency graph itself is stuck (circular or impossible), escalate to user
+
 ### Stall Detection
 
 If a task is `in_progress` longer than `{stall_threshold_minutes}`:
@@ -578,8 +623,8 @@ All agent outputs persist in `{story_output_dir}`: handoff files, reviews, bug r
 ### Step 3: Verify Story Report
 JUDGE writes `story-report.md` as part of its SHIP verdict (Step 14b). Verify the file exists in `{story_output_dir}`. If missing (JUDGE error), write it yourself using the template at `.valent-pipeline/templates/story-report.template.md`.
 
-### Step 4: Tear Down Teammates
-Tear down all per-story teammates. Send `shutdown_request` to each individually.
+### Step 4: Tear Down Heartbeat and Teammates
+Delete the heartbeat and Knowledge cache keep-alive cron jobs using `CronDelete` with their stored job IDs. Then tear down all per-story teammates. Send `shutdown_request` to each individually.
 
 **Knowledge Agent exception:** If `{is_epic_run}` is true, do NOT tear down the Knowledge Agent. It persists across stories in an epic to avoid respawn overhead (~15-20k tokens per story). It will receive a `[STORY-RESET]` at the next story's kick-off. Tear down Knowledge only at epic completion (final story in the epic).
 
@@ -665,14 +710,14 @@ Read each orchestration step file in sequence:
 
 1. `.valent-pipeline/steps/orchestration/sprint-init.md` — compute velocity, resolve candidates, set sprint state
 2. `.valent-pipeline/steps/orchestration/sprint-groom.md` — spawn Phase 1 agents, pipeline stories through REQS → UXA → QA-A → READINESS (assembly-line parallelism), rework loop, index to SQLite
-3. `.valent-pipeline/steps/orchestration/sprint-size.md` — spawn BEND/FEND with estimation step files, assign Fibonacci points, kill estimation agents
+3. `.valent-pipeline/steps/orchestration/sprint-size.md` — spawn BEND/FEND with estimate step first, assign Fibonacci points, agents persist into execution
 4. `.valent-pipeline/steps/orchestration/sprint-plan.md` — greedy packing by priority, write sprint plan + status YAML, validate, kill Phase 1 agents
 5. `.valent-pipeline/steps/orchestration/sprint-execute.md` — execute stories sequentially with budget enforcement, Phase 2 agents per story, update status YAML in real-time
 6. `.valent-pipeline/steps/orchestration/sprint-review.md` — diff planned vs actuals, record calibration data, trigger retrospective, check for next sprint
 
 **Key differences from story mode:**
 - Phase 1 agents (REQS, UXA, QA-A, READINESS) stay alive during grooming batch, killed before execution
-- Phase 2 agents (BEND, FEND, CRITIC, QA-B, JUDGE) killed and respawned per story during execution
+- Phase 2 agents: BEND/FEND persist from sizing into story 1, then killed and respawned fresh for story 2+. CRITIC, QA-B, JUDGE spawned fresh per story.
 - Grooming indexes to `artifacts_working` table; execution queries `artifacts` (main table)
 - Budget enforcement: check cumulative execution time before each story start
 - Retrospective fires at sprint boundary, not story count

package/pipeline/steps/orchestration/sprint-execute.md

@@ -2,7 +2,7 @@
 
 **Condition:** Only execute in sprint mode (`{is_sprint_mode}` is true).
 
-Execute sprint stories sequentially. Phase 2 agents (BEND, FEND, CRITIC, QA-B, JUDGE) are spawned fresh per story and killed after each.
+Execute sprint stories sequentially. BEND/FEND persist from the sizing phase into story 1. CRITIC, QA-B, and JUDGE are spawned fresh for story 1. For story 2+, all Phase 2 agents are spawned fresh and killed after each.
 
 ## For Each Sprint Story (Sequential)
 
@@ -22,7 +22,8 @@ For each story in sprint order:
 1. Update story status to `development` in both `{backlog_path}` and `sprint-{n}-status.yaml`
 2. Execute the standard story flow:
    - Create story branch
-   - Spawn Phase 2 agents per the task graph (BEND, FEND, CRITIC, QA-B, JUDGE)
+   - **Story 1:** BEND/FEND already alive from sizing — spawn only CRITIC, QA-B, JUDGE fresh. Send implementation context to existing BEND/FEND.
+   - **Story 2+:** Spawn all Phase 2 agents fresh per the task graph (BEND, FEND, CRITIC, QA-B, JUDGE)
    - Agents query Knowledge/SQLite for grooming context (NOT in-context from Phase 1)
    - Monitor execution, handle rejections, gate verdicts
    - On JUDGE SHIP: merge branch, record actuals

package/pipeline/steps/orchestration/sprint-size.md

@@ -2,33 +2,36 @@
 
 **Condition:** Only execute in sprint mode (`{is_sprint_mode}` is true).
 
-## Step 1: Spawn Estimation Agents
+## Step 1: Spawn BEND/FEND
 
-Spawn BEND with `.valent-pipeline/steps/bend/estimate.md` step file only (no implementation tools).
+Scan groomed stories' `testing_profiles` in `{backlog_path}`:
 
-If any groomed stories have `fullstack-web` or `frontend-only` surface, also spawn FEND with `.valent-pipeline/steps/fend/estimate.md`.
+- Spawn BEND if any groomed story has `api` or `data-pipeline` in `testing_profiles`
+- Spawn FEND if any groomed story has `ui` in `testing_profiles`
 
-Pass `{estimation_model}` and `{correction_directives}` (calibration directives) in the spawn context.
+Spawn with their normal prompt template and pass `.valent-pipeline/steps/{agent}/estimate.md` as the first step. Pass `{estimation_model}` and `{correction_directives}` (calibration directives) in the spawn context.
+
+These agents persist into execution — they are NOT killed after sizing.
 
 ## Step 2: Size Each Groomed Story
 
 For each story with status `groomed`:
 
 1. Update status to `sizing` in `{backlog_path}`
-2. Send story context (reqs-brief, uxa-spec, qa-test-spec) to BEND
-3. BEND writes `bend-estimation.md` with Fibonacci points
-4. If full-stack: FEND writes `fend-estimation.md` with Fibonacci points
+2. Read story's `testing_profiles` from `{backlog_path}`
+3. Dispatch based on profiles:
+   - `[api]` or `[data-pipeline]`: send story context to BEND only
+   - `[ui]` only: send story context to FEND only
+   - `[api, ui]` (fullstack): send story context to both BEND and FEND
+4. Agents write estimation files (`bend-estimation.md` and/or `fend-estimation.md`)
 5. **Record points:**
-   - Backend-only: `story_points = BEND estimate`
-   - Full-stack: `story_points = BEND estimate + FEND estimate`
+   - Backend-only (`[api]`): `story_points = BEND estimate`
+   - Frontend-only (`[ui]`): `story_points = FEND estimate`
+   - Fullstack (`[api, ui]`): `story_points = BEND estimate + FEND estimate`
    - Data-pipeline: `story_points = BEND estimate`
 6. Update story's `story_points` field in `{backlog_path}`
 
-## Step 3: Kill Estimation Agents
-
-In epic/project mode: kill BEND and FEND after sizing all stories. They will be respawned fresh per story during execution (each story needs clean code context).
-
-## Step 4: Update Sprint State
+## Step 3: Update Sprint State
 
 Update `pipeline-state.json`: `current_sprint.phase = "planning"`.
 

package/skills/valent-run-epic/SKILL.md

@@ -97,7 +97,7 @@ Phase 1 agents (REQS, UXA, QA-A, READINESS) stay alive across the grooming batch
 
 Read and follow `.valent-pipeline/steps/orchestration/sprint-size.md`.
 
-BEND/FEND estimate Fibonacci points per story. Kill estimation agents after sizing.
+BEND/FEND estimate Fibonacci points per story (filtered by `testing_profiles`). Agents persist into story 1 execution.
 
 #### 4f. Sprint Planning
 

package/skills/valent-run-project/SKILL.md

@@ -111,7 +111,7 @@ Phase 1 agents (REQS, UXA, QA-A, READINESS) stay alive across the grooming batch
 
 Read and follow `.valent-pipeline/steps/orchestration/sprint-size.md`.
 
-BEND/FEND estimate Fibonacci points per story. Kill estimation agents after sizing.
+BEND/FEND estimate Fibonacci points per story (filtered by `testing_profiles`). Agents persist into story 1 execution.
 
 #### 4f. Sprint Planning