valent-pipeline 0.7.0 → 0.18.0

package/README.md

@@ -29,27 +29,28 @@ npx valent-pipeline init
 
 ## How It Works
 
-A persistent **Lead** agent reads your story, assembles a team of specialist agents, and orchestrates them through a dependency-driven pipeline:
+A deterministic **Workflow orchestrator** (`sprint.workflow.js`) reads your story, spawns specialist agents per task, and drives them through a dependency-driven pipeline:
 
 ```
-REQS -> UXA -> QA-A -> READINESS -> BEND + FEND -> CRITIC -> QA-B -> JUDGE -> SHIP
+REQS -> UXA -> QA-A -> SPECCHECK -> RED -> BEND + FEND -> STATIC -> CRITIC -> GREEN -> QA-B -> EVIDENCE -> JUDGE -> SHIP
 ```
 
-1. **REQS** translates acceptance criteria into an implementation brief
+1. **REQS** translates acceptance criteria into an implementation brief (+ a machine-readable AC manifest)
 2. **UXA** converts UX specs into component specifications (frontend projects)
-3. **QA-A** writes behavioral test specifications *before any code exists*
-4. **READINESS** gate validates the spec chain -- stops on first failure
+3. **QA-A** writes behavioral test specifications *before any code exists* (+ a spec manifest mapping every case to its AC)
+4. **SPECCHECK** gate validates the spec chain mechanically (`valent spec check` + `trace check`); **RED** gate proves the acceptance suite fails pre-implementation (ATDD)
 5. **BEND + FEND** implement production code and tests in parallel
-6. **CRITIC** runs a 3-pass adversarial code review (blind hunt, edge cases, acceptance audit)
-7. **QA-B** executes tests against real infrastructure, files bugs, builds traceability matrix
-8. **JUDGE** makes an evidence-based SHIP or REJECT decision
-9. **Lead** commits code, writes the story report, and picks the next story
+6. **STATIC** gate runs deterministic checks at mechanical cost: lint/type commands, spec-conformance (no dropped assertion targets or un-waived skips), trace check (every AC has a covering case)
+7. **CRITIC** runs a 3-pass adversarial code review (blind hunt, edge cases, acceptance audit), pinning the git SHA it reviewed
+8. **QA-B** executes tests against real infrastructure **through `evidence run`** — exit codes, full logs, junit reports (frozen + SHA-256-hashed), and the git SHA are captured by the CLI, never transcribed by a model — then files bugs and builds the traceability matrix
+9. **EVIDENCE** gate deterministically re-verifies the machine record: report hashes intact, junit green (with the brownfield pre-existing carve-out), every spec'd case executed, CRITIC's pin matches the tested SHA
+10. **JUDGE** makes the SHIP or REJECT decision from the machine verdicts (its self-reported numbers are recounted from the artifacts afterward — a mismatch stops the run)
 
-Two quality gates (**READINESS** and **JUDGE**) enforce pass/fail checkpoints. Rejection loops send work back to the responsible agent with specific corrections, with a circuit breaker to prevent infinite cycles.
+Quality gates (**SPECCHECK**, **RED**, **STATIC**, **CRITIC**, **GREEN**, **EVIDENCE**, **JUDGE**) enforce pass/fail checkpoints. Rejection loops send work back to the responsible agent with specific corrections, with a code-owned circuit breaker to prevent infinite cycles. The trust property throughout: **facts are captured by code, not transcribed by models** — test results live in `stories/<id>/evidence/` as hashed, machine-written records that gates re-verify.
 
 ## Project Types
 
-The pipeline supports 7 project types, each with a tailored task graph and specialized developer agent:
+The pipeline supports 9 project types, each with a tailored task graph and specialized developer agent:
 
 | Project Type | Developer Agent | Agents Skipped |
 |---|---|---|
@@ -60,9 +61,10 @@ The pipeline supports 7 project types, each with a tailored task graph and speci
 | `mcp-server` | MCP-DEV | UXA, FEND, PMCP |
 | `document-generation` | DOCGEN | UXA, FEND, PMCP |
 | `library` | LIBDEV | UXA, FEND, PMCP |
+| `cli-tool` | CLI-DEV | UXA, FEND, PMCP |
 | `mobile-app` | MOBILE | *(conditional)* |
 
-The Lead selects which agents to spawn based on `project.type` in your `pipeline-config.yaml` and the story's `testing_profiles`.
+The workflow selects which agents to spawn based on `project.type` in your `pipeline-config.yaml` and the story's `testing_profiles` (resolved deterministically by `resolve-graph`).
 
 ## Agent Roster
 
@@ -75,13 +77,14 @@ Spawned fresh per story, torn down after ship or cancel.
 | REQS | Sonnet | Requirements analyst | `reqs-brief.md` |
 | UXA | Sonnet | UX specification | `uxa-spec.md` |
 | QA-A | Sonnet | Test specification | `qa-test-spec.md`, `visual-validation-checklist.md` |
-| READINESS | Sonnet | Spec quality gate | `readiness-review.md` |
+| SPECCHECK | Haiku | Mechanical spec gate (artifact matrix + AC coverage CLIs) | gate transcript |
 | BEND | Sonnet | Backend developer | `bend-handoff.md` |
 | FEND | Sonnet | Frontend developer | `fend-handoff.md` |
 | CRITIC | Opus | Adversarial code reviewer | `critic-review.md` |
 | QA-B | Sonnet | Test executor | `execution-report.md`, `bugs.md`, `traceability-matrix.md` |
-| JUDGE | Sonnet | Final quality gate | `judge-review.md`, `judge-decision.md` |
-| Knowledge | Haiku | Knowledge retrieval | *(inbox only)* |
+| JUDGE | Sonnet/Opus | Final quality gate (evidence pass / binding decision) | `judge-review.md`, `judge-decision.md` |
+
+Knowledge retrieval has no dedicated agent — every agent self-serves from curated files and the SQLite knowledge base via the `db` CLI commands.
 
 ### Domain Developer Agents
 
@@ -92,19 +95,18 @@ Specialized agents that replace BEND for non-API project types:
 | DATA | Sonnet | `data-pipeline` | `data-handoff.md` |
 | MCP-DEV | Sonnet | `mcp-server` | `mcp-dev-handoff.md` |
 | LIBDEV | Sonnet | `library` | `libdev-handoff.md` |
+| CLI-DEV | Sonnet | `cli-tool` | `cli-dev-handoff.md` |
 | DOCGEN | Sonnet | `document-generation` | `docgen-handoff.md` |
 | IAC | Sonnet | Cross-cutting (any type) | `iac-handoff.md` |
 | MOBILE | Sonnet | `mobile-app` | `mobile-handoff.md` |
 
-### Persistent & Ephemeral Agents
+### Cross-Story Stages
 
-| Agent | Model | Lifecycle | Trigger |
-|---|---|---|---|
-| Lead | Opus | Persistent across stories | Always running |
-| PMCP | Sonnet | Ephemeral | QA-B requests visual validation |
-| Embed | Haiku | Ephemeral | After Retrospective curates |
-| Retrospective | Sonnet | Ephemeral | Every N stories (configurable) |
-| Help | Haiku | Ephemeral | User request |
+| Stage | Model | Trigger |
+|---|---|---|
+| PMCP (visual evidence) | Haiku | Orchestrator stage after QA-B on `ui` stories |
+| Retrospective | Sonnet | Retro workflow after a sprint |
+| Knowledge persist | Haiku | `db index-curated` CLI step driven by the retro workflow |
 
 ## Installation
 
@@ -177,13 +179,19 @@ models:
   haiku: [Knowledge, Embed, Help]  # Retrieval, indexing, lookups
 
 quality:
-  max_rejection_cycles: 5          # Circuit breaker for rejection loops
+  max_rejection_cycles: 3          # Circuit breaker for rejection loops
   retrospective_every_n_stories: 5 # Retrospective trigger frequency
   stall_threshold_minutes: 15      # Agent stall detection timeout
 
-git:
-  target_branch: ""                # Base branch for story branches
-  story_branch_prefix: story/      # Branch naming convention
+git:                               # Story git flow (valent git start-story|commit-phase|ship-story|leave-story)
+  enabled: true                    # Branch-per-story + typed commit trail + merge --no-ff on ship
+  target_branch: ""                # Base/merge-back branch ("" = the branch the sprint starts on)
+  story_branch_prefix: story/      # Branch naming convention (story/<id>)
+  parallelism:                     # Concurrent stories in git worktrees (merges stay serialized)
+    enabled: false                 # OFF = strictly sequential sprint (the default)
+    max_stories: 2                 # Concurrency cap when enabled
+    worktree_dir: .valent-worktrees  # Worktrees at <dir>/<story-id> (ignored via .git/info/exclude)
+    setup_commands: []             # Run in each fresh worktree (env files, per-story ports, installs)
 
 knowledge:
   mode: sqlite                     # none | sqlite | local-docker | connect-to-existing
@@ -212,10 +220,17 @@ sprint:                            # Only used in epic/project mode
 | Command | Description |
 |---|---|
 | `valent-pipeline db init` | Initialize SQLite knowledge database |
-| `valent-pipeline db rebuild` | Drop and recreate all tables |
-| `valent-pipeline db index <story-dir>` | Index a story's artifacts |
-| `valent-pipeline db query <text>` | Full-text search across artifacts |
-| `valent-pipeline db embed <file>` | Generate and store embeddings |
+| `valent-pipeline db rebuild` | Rebuild the database from story artifacts |
+| `valent-pipeline db index-handoff --file <path>` | Index a single handoff artifact |
+| `valent-pipeline db search --query <text>` | Full-text search across all artifacts |
+| `valent-pipeline db query-artifact --story <id> --type <type>` | Fetch a specific artifact |
+| `valent-pipeline db query-directives [--agent <role>]` | Get active correction directives |
+| `valent-pipeline db add-directive --json-file <path> --batch <n>` | Persist correction directives (canonical YAML + queryable index, one call) |
+| `valent-pipeline db retire-directive --id <id> [--status expired\|superseded]` | Expire/supersede a directive (status change only — audit trail kept) |
+| `valent-pipeline db sync-directives` | Re-derive the directives index from knowledge/correction-directives.yaml |
+| `valent-pipeline db index-curated --file <path>` | Process embed-instructions.md from a retrospective |
+
+Calibration/query helpers: `db record-calibration`, `db query-velocity`, `db query-list`, `db query-stories`, `db query-bugs-since` (see `valent-pipeline db --help`).
 
 ### Claude Code Skills
 
@@ -228,6 +243,7 @@ Invoked as slash commands inside Claude Code:
 | `/valent-run-story-workflow STORY-ID` | Execute a single story via the Claude Code **Workflow** orchestrator |
 | `/valent-run-epic-workflow EPIC-ID` | Execute an epic (sprint planning + execution) via the **Workflow** orchestrator |
 | `/valent-run-project-workflow` | Execute a full project across all epics via the **Workflow** orchestrator |
+| `/valent-run-spike SPIKE-ID` | Run a time-boxed de-risking spike to a recorded GO/STEP-DOWN/HALT decision |
 | `/valent-run-deferred-tests` | Run deferred iOS tests on Mac |
 | `/valent-debug-export` | Export diagnostic dump |
 | `/valent-help` | Pipeline documentation and FAQ |
@@ -263,7 +279,7 @@ For each story, the pipeline produces 15+ artifacts in `stories/{story-id}/outpu
 | `execution-report.md` | QA-B | Test execution results |
 | `bugs.md` | QA-B | Filed bugs with priorities |
 | `traceability-matrix.md` | QA-B | AC-to-test coverage map |
-| `readiness-review.md` | READINESS | Spec gate results |
+| `evidence/atdd-red.json`, `evidence/proof.json` | RED/GREEN gates (CLI-written) | ATDD red baseline + the red/green/diff proof object |
 | `judge-review.md` | JUDGE | Bug review findings |
 | `judge-decision.md` | JUDGE | Ship/reject decision with evidence |
 | `story-report.md` | orchestrator | Story completion summary |
@@ -274,10 +290,10 @@ Plus committed, tested production code in your project source tree.
 
 All inter-agent communication follows the [Distilled Communication Standard](pipeline/docs/communication-standard.md):
 
-- **Handoff documents** -- structured artifacts with YAML frontmatter, orchestrator summary, and facts-only content. Every handoff follows a [template skeleton](pipeline/docs/template-skeleton.md).
-- **Inbox messages** -- terse coordination messages (~500 tokens max) with file pointers. Types include `[HANDOFF]`, `[BLOCKER]`, `[REVISION]`, `[CRITIC-REJECTION]`, `[BUG]`, `[DESIGN-COUNCIL]`, `[ESCALATION]`.
+- **Handoff documents** -- structured artifacts with YAML frontmatter, orchestrator summary, and facts-only content. Every handoff follows a [template skeleton](pipeline/docs/template-skeleton.md). The handoff file IS the completion signal -- the orchestrator sequences agents from it.
+- **Structured returns** -- each agent returns a terse, schema-validated machine block with file pointers; the orchestrator routes all coordination (rejections, bugs, escalations) from these.
 - **Design Council** -- structured deliberation protocol for contested design decisions with position statements, synthesis, and escalation to user if consensus fails.
-- **Human Escalation** -- when agent deliberation is insufficient, the Lead surfaces the issue to the user with full context.
+- **Human Escalation** -- when agent deliberation is insufficient, the orchestrator surfaces the issue to the user with full context.
 
 ## Knowledge System
 
@@ -287,9 +303,9 @@ The pipeline learns from its own output through a [knowledge system](pipeline/do
 |---|---|---|
 | Correction directives | `knowledge/correction-directives.yaml` | Behavioral changes for agents from past patterns |
 | Curated knowledge | `knowledge/curated/` | Conventions, validated patterns, known pitfalls |
-| SQLite / ChromaDB | `.valent-pipeline/pipeline.db` | Embedding-based retrieval (optional) |
+| SQLite (FTS5) | `.valent-pipeline/pipeline.db` | Full-text retrieval over indexed artifacts and curated lessons |
 
-The **Retrospective** agent (triggered every N stories) is the sole gatekeeper for what enters persistent knowledge. It analyzes batch outputs, writes correction directives, and produces indexing instructions for the **Embed** agent. The **Knowledge** agent reads all sources and responds to teammate queries during story execution.
+The retrospective workflow (`retro.workflow.js`, triggered every N stories) is the sole gatekeeper for what enters persistent knowledge: it analyzes batch outputs, synthesizes gated correction directives, and emits indexing instructions executed by its EMBED stage. During story execution, agents self-serve from the knowledge sources via the `valent-knowledge` skill.
 
 ### Knowledge Modes
 
@@ -302,7 +318,7 @@ The **Retrospective** agent (triggered every N stories) is the sole gatekeeper f
 
 ## Execution Modes
 
-The pipeline runs on a single orchestration path: the **Claude Code Workflow** path. A deterministic Workflow script (`pipeline/orchestrators/claude-code/{plan,sprint,retro}.workflow.js`) drives the pipeline with schema-validated gates, a code-owned rejection cap, parallel CRITIC passes, and journal-based resume (`resumeFromRunId`). Control flow lives in JavaScript and the journal — not in a model interpreting prose. Validated by `scripts/test-workflow.js` but not yet exercised end-to-end against a live story — validate against a fixture before relying on it. See `pipeline/orchestrators/claude-code/README.md`.
+The pipeline runs on a single orchestration path: the **Claude Code Workflow** path. A deterministic Workflow script (`pipeline/orchestrators/claude-code/{plan,sprint,retro}.workflow.js`) drives the pipeline with schema-validated gates, a code-owned rejection cap, parallel CRITIC passes, and journal-based resume (`resumeFromRunId`). Control flow lives in JavaScript and the journal — not in a model interpreting prose. Control flow is validated by `scripts/test-workflow.js`, and the orchestrator is exercised end-to-end against live stories (live runs have driven the version history). See `pipeline/orchestrators/claude-code/README.md`.
 
 > Requires Claude Code (the Workflow tool). `runtime.provider` must be `claude-code`.
 
@@ -340,14 +356,20 @@ Converts your epics and stories documents into a prioritized `pipeline-backlog.y
 
 ## Quality Gates
 
-### READINESS Gate
+### SPECCHECK Gate (mechanical)
+
+Validates the spec chain before any code is written — as CLIs, not LLM judgment:
+- `valent spec check`: artifact-existence matrix per testing profile, acceptance-file existence, and the acceptance-tier mock ban
+- `valent trace check`: every unwaived AC covered by at least one spec'd case
+
+Rework routes to the CLI-named owner (REQS/UXA/QA-A) with downstream specs re-derived automatically.
 
-Validates the spec chain before any code is written:
-- REQS brief completeness and accuracy
-- UXA spec consistency (frontend projects)
-- QA test spec coverage and depth
+### RED / GREEN Gates (ATDD)
 
-Stops on first failure. The responsible upstream agent must rework before the pipeline proceeds.
+When `atdd.command` is configured, QA-A authors **executable acceptance tests** before any implementation exists:
+- **RED** (pre-dev): the suite runs via `valent evidence run` and every required acceptance case must FAIL — a pre-passing test is a spec bug. The passing red writes `evidence/atdd-red.json`, snapshot-hashing the acceptance sources.
+- The acceptance tests are **read-only for dev agents** — any edit is hash-detected and auto-REJECTed to QA-A arbitration (restore, or amend with an audited `evidence rebaseline`).
+- **GREEN** (post-CRITIC): the suite re-runs and must pass with the sources byte-identical since red. `valent evidence proof` then assembles `proof.json`: the failing run before AI wrote code, the passing run after, the exact diff between, and the hashes.
 
 ### JUDGE Gate
 
@@ -358,14 +380,14 @@ Makes the final ship decision based on evidence:
 - PMCP visual evidence (UI projects)
 - Applies "evidence over assertion" -- independently verifies every upstream claim
 
-Verdicts: **SHIP** (commit and close), **SHIP-PARTIAL** (mobile: ship Android, defer iOS), **REJECT** (send back with corrections).
+Verdicts: **SHIP** (the story branch merges `--no-ff` into the target branch and the backlog flips to shipped), **SHIP-PARTIAL** (mobile: ship Android, defer iOS), **REJECT** (the story branch is left unmerged for the fix/retry; send back with corrections).
 
 ### Rejection Loops
 
 When CRITIC or JUDGE rejects work:
 1. Lead re-queues the responsible agent with the specific rejection findings
 2. Agent reworks and resubmits
-3. Circuit breaker (`max_rejection_cycles`, default 5) prevents infinite loops
+3. Circuit breaker (`max_rejection_cycles`, default 3) prevents infinite loops
 4. After max cycles, Lead escalates to user
 
 ## Crash Recovery
@@ -397,7 +419,6 @@ After initialization, the pipeline installs to `.valent-pipeline/` in your proje
     qa-a/                     #   Test spec steps (domain-specific)
     qa-b/                     #   Test execution steps (domain-specific)
     reqs/                     #   Requirements analysis steps
-    readiness/                #   Readiness gate steps
     judge/                    #   Judge gate steps
     orchestration/            #   Shared orchestration steps (config, story resolution, status)
     retrospective/            #   Retrospective analysis steps
@@ -410,7 +431,6 @@ After initialization, the pipeline installs to `.valent-pipeline/` in your proje
     mcp-dev/                  #   MCP server developer steps
     mobile/                   #   Mobile developer steps
     uxa/                      #   UX specification steps
-  spawn-templates/            # Agent spawn configuration
   scripts/                    # Pipeline utility scripts
   docs/                       # Pipeline reference documentation
   knowledge/
@@ -428,19 +448,12 @@ Full reference documentation lives in `pipeline/docs/`:
 | [Pipeline Overview](pipeline/docs/pipeline-overview.md) | Architecture, flow, artifact map |
 | [Agent Reference](pipeline/docs/agent-reference.md) | All agents, models, inputs/outputs |
 | [Communication Standard](pipeline/docs/communication-standard.md) | Handoff format, inbox protocol, Design Council |
-| [Lead Lifecycle](pipeline/docs/lead-lifecycle.md) | Kick-off, monitoring, ship, crash recovery |
 | [Task Graph Specification](pipeline/docs/task-graph.md) | Dependencies, task states, claiming |
 | [Pipeline State Schema](pipeline/docs/pipeline-state-schema.md) | JSON schema for pipeline-state.json |
 | [Knowledge System](pipeline/docs/knowledge-system.md) | RAG assessment, correction directives, curation |
 | [Template Skeleton](pipeline/docs/template-skeleton.md) | Universal handoff document structure |
 | [NPX Packaging](pipeline/docs/npx-packaging.md) | Package distribution and init workflow |
 
-### Reference
-
-| Document | Description |
-|---|---|
-| [Refactor Checklist](pipeline/docs/design/refactor-checklist.md) | Every location to update when changing agents, config, tables, or phases |
-
 ## License
 
 MIT