shipwright-cli 3.2.0 → 3.3.0

package/docs/plans/2026-03-01-pipeline-audit-trail-design.md

@@ -0,0 +1,145 @@
+# Pipeline Audit Trail — Design Document
+
+## Goal
+
+Add structured, compliance-grade audit logging to Shipwright pipelines so every run produces a complete traceability chain: prompt in, plan out, test evidence, audit decisions, and final outcome.
+
+## Problem
+
+Today the pipeline logs raw output (iteration logs, test stdout, pass/fail verdicts) but has critical gaps:
+
+1. **Prompts are discarded** — `compose_prompt()` builds the prompt and passes it directly to `claude -p`. The input is never saved to disk.
+2. **Audit verdicts are opaque** — `audit-iter-N.log` contains ~40 bytes ("verdict · recommendation") with no evidence chain or rationale.
+3. **No cross-stage lineage** — plan.md, design.md, and build output are separate files with no index connecting them.
+4. **Token tracking is incomplete** — `.claude-tokens-*.log` files exist but are often empty.
+
+## Architecture
+
+### Audit Artifacts (per pipeline run)
+
+```
+.claude/pipeline-artifacts/
+├── pipeline-audit.jsonl     # Append-only structured event log (crash-safe)
+├── pipeline-audit.json      # Consolidated summary (generated at end)
+└── pipeline-audit.md        # Human-readable compliance report (generated at end)
+```
+
+### Raw Artifact Preservation
+
+```
+.claude/loop-logs/
+├── iteration-N.prompt.txt   # NEW: Full prompt text saved before sending to Claude
+├── iteration-N.log          # Claude's text response (existing)
+├── iteration-N.json         # Claude's JSON response (existing)
+├── test-evidence-iter-N.json # Structured test results (existing)
+```
+
+### Event Types
+
+| Event | When | Payload |
+|---|---|---|
+| `pipeline.start` | Pipeline begins | Issue, goal, template, model, stages, git SHA |
+| `stage.start` | Stage begins | Stage name, inputs, config |
+| `stage.complete` | Stage ends | Duration, verdict, output artifact paths |
+| `loop.prompt` | Before Claude call | Prompt char count, path to `.prompt.txt` |
+| `loop.response` | After Claude call | Response length, exit code, token usage, path to `.json` |
+| `loop.test_gate` | After test execution | Structured test evidence per command |
+| `loop.audit_verdict` | After audit agent | Verdict, rationale, evidence |
+| `loop.verification_gap` | Verification gap fires | Tests pass + audit disagrees, resolution |
+| `pipeline.complete` | Pipeline ends | Status, duration, cost, PR URL |
+
+### New File: `scripts/lib/audit-trail.sh`
+
+Single library with four functions:
+
+- `audit_init()` — Creates JSONL file, writes `pipeline.start` event
+- `audit_emit(event_type, key=value...)` — Appends one JSON line
+- `audit_save_prompt(prompt_text, iteration)` — Saves `iteration-N.prompt.txt`
+- `audit_finalize()` — Reads JSONL, generates `.json` + `.md` reports
+
+### Integration Points (5 files, ~30 lines total)
+
+| File | Change |
+|---|---|
+| `scripts/lib/pipeline-stages.sh` | Source library, `audit_init` at start, `audit_emit` at stage transitions, `audit_finalize` at end |
+| `scripts/lib/loop-iteration.sh` | `audit_save_prompt` before `claude -p`, `audit_emit` after response |
+| `scripts/sw-loop.sh` | `audit_emit` in `run_test_gate()` and verification gap handler |
+| `scripts/sw-loop.sh` | Source `audit-trail.sh` in imports |
+| `scripts/lib/pipeline-stages.sh` | Source `audit-trail.sh` in imports |
+
+### Fail-Open Principle
+
+All audit calls wrapped in `|| true`. Audit is observability, never a gate. Pipeline continues unaffected if audit logging fails.
+
+## Output Formats
+
+### pipeline-audit.md (Compliance Report)
+
+```markdown
+# Pipeline Audit Report — Issue #192
+
+| Field | Value |
+|---|---|
+| **Pipeline** | standard |
+| **Issue** | #192 |
+| **Model** | sonnet |
+| **Started** | 2026-03-01T14:23:00Z |
+| **Duration** | 24m 18s |
+| **Outcome** | SUCCESS |
+| **PR** | #205 |
+
+## Stage Summary
+
+| Stage | Duration | Verdict | Artifacts |
+|---|---|---|---|
+| intake | 1m 48s | pass | intake.json |
+| plan | 6m 51s | pass | plan.md |
+| build | 8m 22s | pass | 4 iterations |
+
+## Build Loop Detail
+
+### Iteration 1
+- **Prompt**: 48,231 chars → iteration-1.prompt.txt
+- **Response**: 12,440 chars, exit 0
+- **Tests**: npm test (exit 0, 18s), bun test (exit 0, 4s)
+- **Audit**: pass
+
+## Token Usage
+
+| Stage | Input | Output | Cost |
+|---|---|---|---|
+| build | 189,000 | 48,000 | $0.82 |
+| **Total** | **201,000** | **51,400** | **$0.86** |
+```
+
+### pipeline-audit.json (Machine-Readable)
+
+```json
+{
+  "version": "1.0",
+  "pipeline_id": "pipeline-192",
+  "issue": 192,
+  "outcome": "success",
+  "started_at": "2026-03-01T14:23:00Z",
+  "duration_s": 1458,
+  "stages": [...],
+  "tokens": {"input": 201000, "output": 51400, "cost_usd": 0.86},
+  "artifacts": ["plan.md", "design.md"]
+}
+```
+
+All paths relative to `.claude/` for portability.
+
+## Testing Strategy
+
+1. Unit tests in `scripts/sw-lib-audit-trail-test.sh` — emit, save, finalize functions
+2. Integration: run build loop with audit enabled, verify JSONL populated
+3. Regression: existing tests (70/70 detection, 56/61 loop) still pass
+4. E2E: run pipeline, verify `.md` and `.json` reports generated
+
+## Non-Goals
+
+- No external storage (S3, database) — files only
+- No real-time streaming of audit events
+- No retroactive audit of past pipeline runs
+- No audit of non-pipeline (standalone loop) runs initially