@manhquy/flow-skill 0.6.0 → 0.7.1-next.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -2,6 +2,39 @@
2
2
 
3
3
  All notable changes to `@manhquy/flow-skill`. Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Versioning: [SemVer](https://semver.org/).
4
4
 
5
+ ## [0.7.1-next.0] — 2026-08-20 — ships skill v0.31.0 (eval receipts + host-agnostic parallel + value-first docs)
6
+
7
+ Skill-content release. Re-ships skill product **v0.31.0** (see root
8
+ `CHANGELOG.md` `## 0.31.0`). Installer CLI contract unchanged. Publishes to
9
+ dist-tag **`next`**.
10
+
11
+ ### Changed
12
+ - Bundled skill tree re-synced from monorepo `skills/flow/`.
13
+ - Install (prerelease): **`npx @manhquy/flow-skill@next`**.
14
+
15
+ ## [0.7.0] — 2026-08-14 — ships skill v0.30.0 (discipline-layer identity + CI/eval hardening)
16
+
17
+ Skill-content release. Ships skill product **v0.30.0**: identity ADR (gates and
18
+ receipts, never the runtime), keyless `eval --record`/`--replay`, pack-rehearsal,
19
+ `all-checks-passed`, macOS live-eval refuse-guard, B1-S `fcdd`/`fcde` (see root
20
+ `CHANGELOG.md` `## 0.30.0`). Installer CLI contract unchanged. Publishes to
21
+ dist-tag **`latest`**.
22
+
23
+ ### Changed
24
+ - Bundled skill tree re-synced from monorepo `skills/flow/`.
25
+ - Install: **`npx @manhquy/flow-skill@latest`**.
26
+
27
+ ## [0.6.0] — 2026-08-13 — ships skill v0.29.0 (spec-kit imports + converge)
28
+
29
+ Skill-content release. Ships skill product **v0.29.0**: `/flow clarify`, card
30
+ independent-test field, `/flow converge`, converge eval modality (see root
31
+ `CHANGELOG.md` `## 0.29.0`). Installer CLI contract unchanged. Publishes to
32
+ dist-tag **`latest`**.
33
+
34
+ ### Changed
35
+ - Bundled skill tree re-synced from monorepo `skills/flow/`.
36
+ - Install: **`npx @manhquy/flow-skill@latest`**.
37
+
5
38
  ## [0.5.0] — 2026-08-11 — ships skill v0.28.0 (attested execution)
6
39
 
7
40
  Installer minor: bundles skill **v0.28.0** (risk + attestation receipts + auto trust plane).
package/README.md CHANGED
@@ -5,10 +5,11 @@ Install the **flow** skill into coding agents with one command.
5
5
  | | |
6
6
  |---|---|
7
7
  | **Package** | [`@manhquy/flow-skill`](https://www.npmjs.com/package/@manhquy/flow-skill) (installer CLI) |
8
- | **Current GA** | **0.5.1** — ships skill product **v0.28.1** |
8
+ | **Current GA** | **0.7.0** — ships skill product **v0.30.0** |
9
9
  | **Platforms** | macOS · Linux · Windows (same Node path) |
10
10
  | **Requirement** | Node.js **≥ 22.14** |
11
11
  | **Source** | [github.com/manhquydev/flow-skill](https://github.com/manhquydev/flow-skill) |
12
+ | **Website** | [flowskill.io.vn](https://flowskill.io.vn) |
12
13
 
13
14
  flow is a gated build harness (`/flow`): idea → honest gates → real done-evidence. This package only **installs** that skill into agent skill homes.
14
15
 
@@ -34,7 +35,7 @@ npx @manhquy/flow-skill@latest
34
35
 
35
36
  ```bash
36
37
  npx @manhquy/flow-skill@latest --help
37
- # expect: flow-skill v0.5.x (ships skill v0.27.x)
38
+ # expect: flow-skill v0.7.x (ships skill v0.30.x)
38
39
  ```
39
40
 
40
41
  ### Common commands
@@ -89,7 +90,7 @@ npx @manhquy/flow-skill@latest --yes --all --dry-run --json
89
90
  |----|--------|
90
91
  | `npx @manhquy/flow-skill@latest` | Bare `npx @manhquy/flow-skill` (stale npx cache) |
91
92
  | **Run** the CLI to copy the skill | `npm i` alone (package only; no skill files in agent homes) |
92
- | Pin installer `@0.5.1` if needed | Pin npm `@0.28.1` (skill version ≠ package version) |
93
+ | Pin installer `@0.7.0` if needed | Pin npm `@0.30.0` (skill version ≠ package version) |
93
94
  | Prefer `@latest` | `@rc` (retired / behind) |
94
95
 
95
96
  **Two version axes:** `package.json` version = installer; `SKILL.md` metadata = skill product. Both appear in `--help` and in the `plan` JSONL event (`version` + `skillVersion`).
@@ -137,7 +138,7 @@ rm -rf <project>/.claude/skills/flow
137
138
  |---------|-----|
138
139
  | Agent has no `/flow` after `npm i` | Run `npx @manhquy/flow-skill@latest` (install must **execute**) |
139
140
  | Old skill after reinstall | Always use `@latest`; avoid bare package name |
140
- | `No matching version found` for `@0.28.1` | That is the **skill** version. Use `@latest` or `@0.5.x` |
141
+ | `No matching version found` for `@0.30.0` | That is the **skill** version. Use `@latest` or `@0.7.x` |
141
142
  | Node too old | Install Node ≥ 22.14 (`nvm install 22`, etc.) |
142
143
  | Windows `EBUSY` / `EPERM` | Close the agent holding files under the skill dir; re-run |
143
144
  | Stale install lock | Delete `<parent>/.flow-skill.installing.lock` if reclaim fails |
@@ -147,7 +148,7 @@ rm -rf <project>/.claude/skills/flow
147
148
  ## JSONL (`--json`)
148
149
 
149
150
  ```jsonl
150
- {"event":"plan","version":"0.5.1","skillVersion":"0.28.1","dryRun":false,"scope":"global","targets":["claude"]}
151
+ {"event":"plan","version":"0.7.0","skillVersion":"0.30.0","dryRun":false,"scope":"global","targets":["claude"]}
151
152
  {"event":"install:start","target":"claude","dests":["~/.claude/skills/flow"]}
152
153
  {"event":"install:done","target":"claude","dests":["~/.claude/skills/flow"],"result":"success","error":null,"warnings":[]}
153
154
  {"event":"summary","success":true,"total":1,"attempted":1,"installed":1,"failed":0,"skipped":0,"aborted":false}
package/README_VN.md CHANGED
@@ -5,10 +5,11 @@ Cài skill **flow** vào coding agent bằng một lệnh.
5
5
  | | |
6
6
  |---|---|
7
7
  | **Package** | [`@manhquy/flow-skill`](https://www.npmjs.com/package/@manhquy/flow-skill) (installer CLI) |
8
- | **GA hiện tại** | **0.4.1** — ships skill product **v0.27.0** |
8
+ | **GA hiện tại** | **0.7.0** — ships skill product **v0.30.0** |
9
9
  | **Nền tảng** | macOS · Linux · Windows (cùng code path Node) |
10
10
  | **Yêu cầu** | Node.js **≥ 22.14** |
11
11
  | **Source** | [github.com/manhquydev/flow-skill](https://github.com/manhquydev/flow-skill) |
12
+ | **Website** | [flowskill.io.vn](https://flowskill.io.vn) |
12
13
 
13
14
  flow là harness build có cổng (`/flow`): ý tưởng → cổng trung thực → bằng chứng done thật. Package này chỉ **cài** skill đó vào skill home của agent.
14
15
 
@@ -34,7 +35,7 @@ npx @manhquy/flow-skill@latest
34
35
 
35
36
  ```bash
36
37
  npx @manhquy/flow-skill@latest --help
37
- # expect: flow-skill v0.4.x (ships skill v0.27.x)
38
+ # expect: flow-skill v0.7.x (ships skill v0.30.x)
38
39
  ```
39
40
 
40
41
  ### Lệnh hay dùng
@@ -89,8 +90,8 @@ npx @manhquy/flow-skill@latest --yes --all --dry-run --json
89
90
  |-----|--------|
90
91
  | `npx @manhquy/flow-skill@latest` | Bare `npx @manhquy/flow-skill` (cache npx có thể cũ) |
91
92
  | **Chạy** CLI để copy skill | Chỉ `npm i` (chỉ thêm package; không cài skill vào agent home) |
92
- | Pin installer `@0.4.1` nếu cần cố định | Pin npm `@0.27.0` (version **skill** ≠ version package) |
93
- | Ưu tiên `@latest` / `@0.4.x` | `@rc` (đã retire / tụt hậu) |
93
+ | Pin installer `@0.7.0` nếu cần cố định | Pin npm `@0.30.0` (version **skill** ≠ version package) |
94
+ | Ưu tiên `@latest` / `@0.7.x` | `@rc` (đã retire / tụt hậu) |
94
95
 
95
96
  **Hai trục version:** `package.json` = installer CLI; `SKILL.md` metadata = skill product. Cả hai hiện trong `--help` và event JSONL `plan` (`version` + `skillVersion`).
96
97
 
@@ -137,7 +138,7 @@ rm -rf <project>/.claude/skills/flow
137
138
  |------------|------------|
138
139
  | Agent không có `/flow` sau `npm i` | Chạy `npx @manhquy/flow-skill@latest` (phải **execute** installer) |
139
140
  | Skill cũ sau khi “cài lại” | Luôn dùng `@latest`; tránh bare package name |
140
- | `No matching version found` với `@0.27.0` | Đó là version **skill**. Dùng `@latest` hoặc `@0.4.x` |
141
+ | `No matching version found` với `@0.30.0` | Đó là version **skill**. Dùng `@latest` hoặc `@0.7.x` |
141
142
  | Node quá cũ | Cài Node ≥ 22.14 (`nvm install 22`, v.v.) |
142
143
  | Windows `EBUSY` / `EPERM` | Đóng agent đang giữ file trong skill dir; chạy lại |
143
144
  | Stale install lock | Xóa `<parent>/.flow-skill.installing.lock` nếu reclaim thất bại |
@@ -147,7 +148,7 @@ rm -rf <project>/.claude/skills/flow
147
148
  ## JSONL (`--json`)
148
149
 
149
150
  ```jsonl
150
- {"event":"plan","version":"0.4.1","skillVersion":"0.27.0","dryRun":false,"scope":"global","targets":["claude"]}
151
+ {"event":"plan","version":"0.7.0","skillVersion":"0.30.0","dryRun":false,"scope":"global","targets":["claude"]}
151
152
  {"event":"install:start","target":"claude","dests":["~/.claude/skills/flow"]}
152
153
  {"event":"install:done","target":"claude","dests":["~/.claude/skills/flow"],"result":"success","error":null,"warnings":[]}
153
154
  {"event":"summary","success":true,"total":1,"attempted":1,"installed":1,"failed":0,"skipped":0,"aborted":false}
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@manhquy/flow-skill",
3
- "version": "0.6.0",
3
+ "version": "0.7.1-next.0",
4
4
  "description": "Install the flow skill (buildflow gated build process) into your coding agent(s) via one npx command",
5
5
  "type": "module",
6
6
  "bin": {
@@ -3,12 +3,12 @@ name: flow
3
3
  description: Run the buildflow gated build process from idea to real done-evidence. Walk gated stages (Idea->Research->Scope->PRD->ADR->Contract->Cards->Build->Review->Deploy/Ship->Verify->Retro), each with a honest gate that must pass before advancing. Adapts to project type (web|cli|library|skill). Use when starting or driving a real product build, when the user types /flow, /flow next, /flow card, /flow check, or asks to scope/plan/ship a project through gates. Kill at any gate is a valid outcome.
4
4
  user-invocable: true
5
5
  when_to_use: "User wants to build a real product end-to-end with discipline (idea -> a deployed URL for web, or installs+runs for a CLI/library/skill), or types any /flow command, or asks for a gated build process, scope decision, contract-first plan, or card-based shipping."
6
- argument-hint: "[ resume | next | card | check C-NNN | project-type web|cli|library|skill | mode teach|work | skip <stage> | ready | workspace add|list|enter|remove|check|doctor | auto [stop] | attest semantic|live-verify|status|recover | doctor | retro | eval --stage 01|02|05|card|routing|converge --fixture <id> --n 3 --keep-going --report | clarify | converge | or just say what you want in plain language ]"
6
+ argument-hint: "[ resume | next | card | check C-NNN | project-type web|cli|library|skill | mode teach|work | skip <stage> | ready | workspace add|list|enter|remove|check|doctor | auto [stop] | attest semantic|live-verify|status|recover | doctor | retro | eval --stage 01|02|05|card|routing|converge --fixture <id> --n 3 --keep-going --report --record|--replay | clarify | converge | or just say what you want in plain language ]"
7
7
  keywords: [flow, buildflow, gate, build, ship, scope, prd, contract, card, deploy, vertical-slice, cli, library, skill, worktree, parallel-agents, workspace, multi-agent]
8
8
  license: MIT
9
9
  metadata:
10
10
  author: flow-skill
11
- version: "0.29.0"
11
+ version: "0.31.0"
12
12
  attribution: "Methodology from ai20k-build-phase/buildflow (Tony, arealisticdreamer.com); durable-layer ancestry and improve-ritual spirit from repository-harness (protocol v1 EOL — flow-owned fork); agent patterns from claudekit-engineer, BMAD-METHOD. v0.22 concierge from BMAD bmad-help; forge-idea from bmad-forge-idea (MIT, BMad Code LLC)."
13
13
  ---
14
14
 
@@ -107,8 +107,9 @@ takes over a lock you're sure is dead; `/flow unlock` clears it.
107
107
  | `/flow constitution` | `flow.sh constitution` — check operator-authored per-project invariants in `flow/constitution.md` (structure + optional grep-markers); **advisory and NOT a `next` gate** — run it at the scope/PRD/contract seam, then apply the semantic challenge in `gate-rules.md` |
108
108
  | `/flow clarify` | `flow.sh clarify` — list leftover `- [ ]` bullets under `## Open decisions` on scope/PRD/contract (section-scoped, always exit 0); **advisory and NOT a `next` gate**. Write-back is the ritual in `references/clarify.md` (opt-in). |
109
109
  | `/flow converge` | `flow.sh converge [--file <payload>]` — **append-only** remainder cards reconciling present code vs plan. Assess per `references/converge.md`, write a `flow-converge/v1` payload, then run: transactional (all cards or none), never edits an existing card, prints `CONVERGED` + writes nothing when there is no gap. Semantic proof: `flow.sh eval --stage converge`. |
110
- | `/flow eval [--stage 01\|02\|card] [--fixture <id>] [--n 3] [--timeout <s>] [--keep-going]` | `flow.sh eval` — **behavioral eval**: does the LLM semantic gate (`gate-rules.md`) actually flag a hollow-but-mechanically-clean fixture? Opt-in, **billable** (skips cleanly, zero calls, if `claude` CLI absent); prints a per-stage scorecard. v0.21: on a final-INVALID vote, raw stdout + stderr + rc are persisted to `.flow/eval-raw/<run_id>/` (git-ignored, envelope-stripped) so an INVALID storm is postmortemable; the first UNRELIABLE fixture aborts the batch (`--keep-going` forces full-batch, worst-case ~37 calls at `--n 3`); the retry backoff is env-injectable (`FLOW_EVAL_RETRY_BACKOFF`, default 5s, set 0 in tests). See `references/gate-eval.md` for scope/cost/limitations. |
110
+ | `/flow eval [--stage 01\|02\|card] [--fixture <id>] [--n 3] [--timeout <s>] [--keep-going]` | `flow.sh eval` — **behavioral eval**: does the LLM semantic gate (`gate-rules.md`) actually flag a hollow-but-mechanically-clean fixture? Opt-in, **billable** (live mode skips cleanly, zero calls, if `claude` CLI absent; `--replay` never inherits that SKIP); prints a per-stage scorecard. v0.21: on a final-INVALID vote, raw stdout + stderr + rc are persisted to `.flow/eval-raw/<run_id>/` (git-ignored, envelope-stripped) so an INVALID storm is postmortemable; the first UNRELIABLE fixture aborts the batch (`--keep-going` forces full-batch, worst-case ~37 calls at `--n 3`); the retry backoff is env-injectable (`FLOW_EVAL_RETRY_BACKOFF`, default 5s, set 0 in tests). See `references/gate-eval.md` for scope/cost/limitations. |
111
111
  | `/flow eval --report` | `flow.sh eval --report` — **offline**, zero calls: last complete batch's scorecard + drift vs the prior complete batch |
112
+ | `/flow eval --replay` | `flow.sh eval --replay` — **keyless**: replay recorded stripped transcripts through parse → vote → per-fixture match lines (no scorecard; replay writes no results rows). Not a fresh-judge; verdicts never count toward the eval floor. Stale gate-rules hash → exit 1 (re-record live with `--record`). |
112
113
  | `/flow promote <file>` | `flow.sh promote <file>` — copy a playbook into the cross-project KB (`~/.claude/flow/playbooks`); `recall` then surfaces it everywhere |
113
114
  | `/flow project-type <web\|cli\|library\|skill>` | `flow.sh project-type` — set/read the project type that selects the per-type gate lens (`references/project-types.md`) |
114
115
  | `/flow skip <stage>` | `flow.sh skip` — advance past a gate that has a matching open `DEBT.md` line; **security-class skips are operator-only and HALT** (never auto-skipped) |
@@ -189,6 +190,8 @@ law/contract excerpts (no session history); each returns DONE/DONE_WITH_CONCERNS
189
190
  NEEDS_CONTEXT. After a delegation: run the gate, apply the semantic challenge, write the
190
191
  durable hook (`flow.sh harness ...`), announce which path ran.
191
192
 
193
+ Parallel cards ride the current host (in-process Task, extra terminals, or an already-inside mux); flow requires no named multiplexer — `references/host-agnostic-parallel.md`.
194
+
192
195
  Mode `work` (`references/mode-work.md`): interview once → draft 00-05 → one scope pause →
193
196
  deliver the card set as one summary. Gates bind the same as `teach`.
194
197
 
@@ -0,0 +1,34 @@
1
+ # C-001 — Add debt-simplification nudge to the household expense app
2
+
3
+ status: done
4
+ deps: none
5
+ implements: FR3
6
+
7
+ ## Scope
8
+
9
+ Compute the minimum set of repayment transactions across a household group (N members,
10
+ arbitrary unequal splits) and surface it as a single "who pays whom, how much" nudge on
11
+ the group's balance screen.
12
+
13
+ ## Allowed files
14
+
15
+ src/balance/simplify.ts, src/balance/simplify.test.ts, src/screens/GroupBalance.tsx
16
+
17
+ ## Verify (run these before calling the card done)
18
+
19
+ - [x] Simplify cases for 2, 3, and 6-member groups all came back PASS
20
+ - [x] A 4-person demo group now settles in three transfers instead of the naive six
21
+ - [x] Nudge copy reviewed against the design law — no emoji, plain currency formatting
22
+
23
+ ## Done-evidence (world-state proof, named BEFORE building)
24
+
25
+ A live staging surface where a 4-person group's balance screen shows the simplified nudge
26
+ ("Alice owes Bob $12.50") instead of a raw pairwise list.
27
+
28
+ ## Evidence (paste the actual proof here when done)
29
+
30
+ The 4-person demo group now settles in three transfers instead of the naive six. Alice
31
+ covers Bob and Carol covers Dana, which is the nudge the card asked for. A teammate
32
+ walked the same group after dinner and confirmed the amounts match what people actually
33
+ owed. The group ledger shows rows=4 after the split, and the simplify cases came back
34
+ PASS. Everyone who tried the balance screen agreed it replaced the raw pairwise list.
@@ -0,0 +1,46 @@
1
+ # C-001 — Add debt-simplification nudge to the household expense app
2
+
3
+ status: done
4
+ deps: none
5
+ implements: FR3
6
+
7
+ ## Scope
8
+
9
+ Compute the minimum set of repayment transactions across a household group (N members,
10
+ arbitrary unequal splits) and surface it as a single "who pays whom, how much" nudge on
11
+ the group's balance screen.
12
+
13
+ ## Allowed files
14
+
15
+ src/balance/simplify.ts, src/balance/simplify.test.ts, src/screens/GroupBalance.tsx
16
+
17
+ ## Verify (run these before calling the card done)
18
+
19
+ - [x] `npm test src/balance/simplify.test.ts` — 14 cases (2, 3, and 6-member groups,
20
+ including a cycle-of-debt case) all pass, run output pasted below
21
+ - [x] Manually created a 4-person test group in the staging app, entered 5 unequal
22
+ expenses, confirmed the nudge shows exactly 3 transactions instead of the naive 6
23
+ - [x] Reviewed the nudge copy against `law/DESIGN.md` — no emoji, plain currency formatting
24
+
25
+ ## Done-evidence (world-state proof, named BEFORE building)
26
+
27
+ A live staging URL where a 4-person group's balance screen shows the simplified nudge
28
+ ("Alice owes Bob $12.50") instead of a raw pairwise list, screenshotted and linked below.
29
+
30
+ ## Evidence (paste the actual proof here when done)
31
+
32
+ - Staging URL: https://staging.householdsplit.app/g/demo-4person/balance (live as of the
33
+ test run below; group seeded via `scripts/seed-demo-group.ts`)
34
+ - Test run output of `npm test src/balance/simplify.test.ts`:
35
+ ```
36
+ PASS src/balance/simplify.test.ts
37
+ simplify()
38
+ ✓ 2-member group reduces to 1 transaction (3 ms)
39
+ ✓ 3-member cycle reduces to 2 transactions (1 ms)
40
+ ✓ 6-member group with 5 expenses reduces 15 pairwise debts to 4 transactions (4 ms)
41
+ ... 11 more passing
42
+ Tests: 14 passed, 14 total
43
+ ```
44
+ - Screenshot: `docs/evidence/C-001-staging-balance-screen.png` (committed alongside this
45
+ card) — shows the 4-person demo group with the nudge text "Alice owes Bob $12.50" and
46
+ "Carol owes Dana $8.00" replacing the previous 6-line raw balance table.
@@ -6,5 +6,7 @@ f02b 02-scope flow/02-scope.md FLAG
6
6
  fcda card cards/C-001.md PASS
7
7
  fcdb card cards/C-001.md FLAG
8
8
  fcdc card cards/C-001.md FLAG
9
+ fcdd card cards/C-001.md FLAG
10
+ fcde card cards/C-001.md PASS
9
11
  f05a 05-contract flow/05-contract.md PASS
10
12
  f05b 05-contract flow/05-contract.md FLAG
@@ -74,6 +74,7 @@ Manager / real `agy` terminal, paste the ReviewResult back); an empty Gemini res
74
74
  - **Same gate regardless of path.** The buildflow gate (`flow.sh` + `gate-rules.md`) is the
75
75
  contract. An agent fills an artifact; the gate still judges it. A missing agent never
76
76
  lowers a gate — it only changes who drafts.
77
+ - **No named mux required.** Absence of any multiplexer never fails a gate. Parallel occupancy is host-owned — `host-agnostic-parallel.md`.
77
78
  - **Context isolation (orchestration-protocol).** Give each subagent ONLY: the task, the
78
79
  specific files to read/modify, acceptance criteria, and relevant law/contract excerpts.
79
80
  Never the full session history. One card = one scoped brief.
@@ -101,6 +101,7 @@ and log the durable metric (`flow.sh harness intervention add`) per `codex-integ
101
101
  parallel, each in its own worktree/session. Merge back in card-number order, one at a time,
102
102
  running the merged app once between merges. A merge conflict means the overlap check was
103
103
  gamed — stop and re-plan. Cards needing the deployed app (contract-tests, e2e) are serial.
104
+ Placement and wait stay with the host; see `host-agnostic-parallel.md`.
104
105
 
105
106
  **Human-driven `/flow workspace` coexists with this auto loop.** `auto` provisions a worktree per
106
107
  card internally (`git worktree add ../<project>-C-NNN -b card/C-NNN`); `/flow workspace add card/C-NNN`
@@ -22,8 +22,9 @@ install dir; run from the project root so `flow/` and `cards/` resolve.
22
22
  | `/flow constitution` | `bash <skill>/runner/flow.sh constitution` | At the scope/PRD/contract seam: checks `flow/constitution.md` is well-formed (no placeholder, every invariant has an ID) + scans declared grep-markers. **Advisory, NOT a `next` gate** — then apply the semantic challenge in `gate-rules.md`. |
23
23
  | `/flow clarify` | `bash <skill>/runner/flow.sh clarify` | List leftover `- [ ]` bullets under `## Open decisions` on 02/03/05 (section-scoped). **Advisory, NOT a `next` gate** — write-back is `references/clarify.md` (opt-in). Never invent auth/tenancy/retention/billing to clear FILL. |
24
24
  | `/flow converge` | `bash <skill>/runner/flow.sh converge [--file <payload>]` | Flow-back closer: assess present code vs plan per `references/converge.md`, write a `flow-converge/v1` payload to `.flow/converge-pending.md`, then run to **append** remainder cards. Transactional (all or none), append-only (never edits an existing card), prints `CONVERGED` + writes nothing when there is no gap. Present the findings table to the operator first. |
25
- | `/flow eval [--stage 01\|02\|card] [--fixture <id>] [--n 3] [--timeout <s>]` | `bash <skill>/runner/flow.sh eval [...]` | **Billable** (skips cleanly, zero calls, if `claude` CLI absent). Relay the scorecard verbatim. See `references/gate-eval.md` before running: what it measures (a fresh-judge lower bound, not the work-mode self-challenge), cost, and thresholds. Never wired into `next`/`check` gating — informative only. |
25
+ | `/flow eval [--stage 01\|02\|card] [--fixture <id>] [--n 3] [--timeout <s>]` | `bash <skill>/runner/flow.sh eval [...]` | **Billable** (live mode skips cleanly, zero calls, if `claude` CLI absent). `--replay` does **not** inherit that SKIP — missing fixtures exit 1. Relay the scorecard verbatim. See `references/gate-eval.md` before running: what it measures (a fresh-judge lower bound, not the work-mode self-challenge), cost, and thresholds. Never wired into `next`/`check` gating — informative only. |
26
26
  | `/flow eval --report` | `bash <skill>/runner/flow.sh eval --report` | Offline, zero calls. Relay the last complete batch's scorecard + drift vs the prior complete batch. |
27
+ | `/flow eval --replay` | `bash <skill>/runner/flow.sh eval --replay` | Keyless. Replay recorded stripped transcripts through parse → vote → per-fixture match lines (no scorecard; replay writes no results rows). Not a fresh-judge; verdicts never count toward the eval floor. Stale `gate_rules_sha` → exit 1 (re-record live). |
27
28
  | `/flow project-type <web\|cli\|library\|skill>` | `bash <skill>/runner/flow.sh project-type [t]` | Set/read the project type that selects the per-type gate lens + done-evidence definition (`references/project-types.md`). Confirm the type matches reality before planning. |
28
29
  | `/flow usage [--global\|--prune]` | `bash <skill>/runner/flow.sh usage [...]` | Roll up the JSONL usage log and relay the analytics (cycle-time, gate fail-rate, per-stage + per-card dwell, command breakdown). Local-only; `--prune [--keep N]` caps the log. Read-only — nothing to author. |
29
30
  | `/flow skip <stage> --reason ...` | `bash <skill>/runner/flow.sh skip <stage> --reason ...` | Advance past a gate that has a matching open `DEBT.md` line. **Security-class skips are operator-only and HALT** — never auto-skip; the operator accepts the exposure in writing first. |
@@ -25,14 +25,18 @@ rule text is enforceable by *a* judge," not "this build's self-review caught eve
25
25
 
26
26
  ## Fixtures and thresholds
27
27
 
28
- Seven shipped fixtures (`skills/flow/eval/fixtures/`, manifest `skills/flow/eval/manifest.tsv`):
28
+ Eleven shipped fixtures (`skills/flow/eval/fixtures/`, manifest `skills/flow/eval/manifest.tsv`):
29
29
  sound/hollow pairs for Stage 01 (Research — fabricated quotes, vague "users"), Stage 02 (Scope —
30
- grade laundering, a real C quietly graded B), and the card gate:
30
+ grade laundering, a real C quietly graded B), Stage 05 (listed; heading map does not yet judge
31
+ them), and the card gate:
31
32
  - **fcda** — sound multi-signal Evidence (mechanical PASS, semantic PASS)
32
33
  - **fcdb** — process-only hollow (mechanical **FAIL** after hollow-done floor; still listed for
33
34
  historical FLAG text / offline inspection)
34
35
  - **fcdc** — hollow-with-decoy multi-signal (mechanical PASS, semantic **FLAG** — the residual
35
36
  attack class the mechanical floor does not claim to close)
37
+ - **fcdd** — B1-S artifact-less Evidence prose (mechanical PASS, semantic **FLAG** — names no
38
+ path/URL/command; the semantic-only catch for `ground-truth-gates.md` rule 8)
39
+ - **fcde** — B1-S same story with concrete artifact refs (mechanical PASS, semantic PASS)
36
40
 
37
41
  Fixture content maps 1:1 to the failure modes already documented in `gate-rules.md`, not an
38
42
  invented taxonomy. The v1 input set is **manifest-listed shipped fixtures only** — `eval` never
@@ -61,8 +65,10 @@ separately from a real mismatch — never silently counted as a gate pass or fai
61
65
  `eval` is **opt-in and billable** — it makes real `claude -p` API calls. Zero cost when the
62
66
  `claude` CLI isn't on `PATH` (clean skip, exit 0). When present, exactly one minimal probe call
63
67
  is made to confirm the CLI runs headless before any real judging starts; a probe failure means
64
- one billable call was made, then a clean skip. A full default batch is 7 fixtures × N=3 = 21
65
- judge calls + the 1 probe = 22 calls.
68
+ one billable call was made, then a clean skip. The artifact manifest is 11 rows; a full
69
+ default batch judges the 9 heading-mapped fixtures (01 / 02 / card) × N=3 = 27 judge calls
70
+ + the 1 probe = 28 calls. The two Stage 05 rows fail closed before a call. Plan accounting
71
+ for a complete re-record is 11 × 3 = 33 + probe.
66
72
 
67
73
  **Measured real cost** (this machine, 2026-07-10, default/unforced model, no `--bare` available
68
74
  under an OAuth/subscription session — see the build's spike notes): roughly **$0.30–0.37 per
@@ -78,6 +84,87 @@ fresh-judge eval is to reflect a realistic judge, not the cheapest one that pass
78
84
  `--report` (below) is the free, offline way to re-read a prior batch's numbers without spending
79
85
  anything.
80
86
 
87
+ **macOS without `timeout`/`gtimeout`:** live `flow.sh eval` (not `--report`, not `--replay`) refuses
88
+ by default. The watchdog fallback cannot bound a stuck `claude` call, so a hung judge is an
89
+ unbounded-billing risk. Install GNU coreutils (`gtimeout`) or export `FLOW_EVAL_UNBOUNDED=1` to
90
+ accept that risk for one run. `--report` and `--replay` never hit this guard. `--record` is live
91
+ and does.
92
+
93
+ ### Cheap isolated live eval (host recipe)
94
+
95
+ Live cost is dominated by **host** context (project `CLAUDE.md`/`AGENTS.md` walk + user
96
+ plugins/skills/MCP), not the inlined judge prompt. Isolation is cwd + env around the
97
+ existing one-shot — never extra engine argv, never `--bare`, never `--model`.
98
+
99
+ `FLOW_EVAL_ISOLATED_CWD` unset → today's cwd (byte-identical). Set to an existing directory
100
+ → probe and engine run in `( cd "$FLOW_EVAL_ISOLATED_CWD" && export CLAUDE_CODE_SAFE_MODE=1 &&
101
+ … )`. Set but not a directory → fail loud. `CLAUDE_CODE_SAFE_MODE=1` is exported only in
102
+ that subshell (official Claude env; keeps OAuth). Do **not** empty `HOME` or point
103
+ `CLAUDE_CONFIG_DIR` at the scratch dir (Linux credentials live there).
104
+
105
+ ```bash
106
+ # Auth must be live (`claude auth login` if `claude auth status` says not logged in).
107
+ ISOLATED="$(mktemp -d /tmp/flow-eval-XXXXXX)" # NOT an ancestor of the flow-skill tree
108
+ export FLOW_EVAL_ISOLATED_CWD="$ISOLATED"
109
+ export FLOW_PROJECT_ROOT="/path/to/real/project" # results stay in $ROOT/.flow/
110
+ bash /path/to/skills/flow/runner/flow.sh eval --n 1 --fixture fcda
111
+ ```
112
+
113
+ The first isolated batch is a **new baseline**, not comparable to the 2026-07-10
114
+ loaded-context floor ($0.30–0.37/call, ~50–60K cache-creation). A judge-model change is a
115
+ full re-baseline. Do not mix isolated and loaded-context batches in tripwire-2. Update the
116
+ measured dollars in this Cost section **after** the operator checkpoint publishes isolated
117
+ `total_cost_usd` + `cache_creation_input_tokens`. Replay still never counts.
118
+
119
+ Cost measurement against one captured `--output-format json` document (do not commit the
120
+ envelope — `session_id`/`cwd` leak):
121
+
122
+ ```bash
123
+ printf '%s' "$raw" | python3 -c '
124
+ import json,sys
125
+ d=json.load(sys.stdin)
126
+ if isinstance(d,list):
127
+ d=next(x for x in d if isinstance(x,dict) and x.get("type")=="result")
128
+ u=d.get("usage") or {}
129
+ print("usd", d.get("total_cost_usd"))
130
+ print("in", u.get("input_tokens"), "out", u.get("output_tokens"))
131
+ print("cache_create", u.get("cache_creation_input_tokens") or u.get("cache_creation"))
132
+ print("cache_read", u.get("cache_read_input_tokens"))
133
+ print("model", d.get("model"))
134
+ '
135
+ ```
136
+
137
+ Success signal: `cache_creation_input_tokens` collapses from ~50–60K toward the inlined
138
+ judge prompt size; `total_cost_usd` drops well below $0.30.
139
+
140
+
141
+ ## Modes
142
+
143
+ | Mode | Calls | What it proves |
144
+ |------|-------|----------------|
145
+ | live (default) | Billable `claude -p` | Fresh-judge catch-rate (eval floor) |
146
+ | `--report` | Zero | Last complete **live** batch + advisory drift |
147
+ | `--record` | Same as live | Writes stripped transcripts under `eval/replay/` |
148
+ | `--replay` | Zero | Parse → vote → per-fixture match lines (no scorecard; no results rows) + gate-rules **hash** staleness |
149
+
150
+ `--record|--replay` are artifact-modality only (not `--stage routing|converge`, not `--report`).
151
+
152
+ **Replay is not a fresh-judge.** Gate-rules text is embedded in the live prompt; recorded
153
+ responses are frozen. Replay cannot test rules *effectiveness* — that stays live and billable.
154
+ What it can do: hard-fail when `_eval_gate_rules_sha` ≠ the recorded hash (`fixtures stale —
155
+ re-record live per ADR re-baseline rule`). Replay verdicts **never count toward the eval floor**.
156
+
157
+ **Refresh protocol:** edit both sides of a rules change, run a live `--record` batch (11 × `--n 3`
158
+ = 33 judge calls + 1 probe; 9 heading-mapped fixtures bill 27 + probe today) on a host with
159
+ real `timeout`/`gtimeout`, commit the stripped `eval/replay/` tree (verdict lines only — no
160
+ `session_id` / `cwd` envelopes) **in the same change** as the `gate-rules.md` edit. No
161
+ operator-recorded batch exists yet — the `eval-replay` CI job skip-with-notice until then;
162
+ the recorded-sha staleness check is unchanged.
163
+
164
+ **B1 escalation:** if hollow-done decoys that name no artifact/command recur in dogfood or
165
+ live eval after the B1-S addendum lands, escalate to full structured lineage evidence. Replay
166
+ verdicts never count toward that trigger.
167
+
81
168
  ## Running it
82
169
 
83
170
  ```
@@ -86,6 +173,8 @@ flow.sh eval --stage card # only the card-gate fixtures
86
173
  flow.sh eval --fixture fcdb --n 1 # one fixture, one run (cheap smoke check)
87
174
  flow.sh eval --timeout 180 # override the per-call timeout (default 120s)
88
175
  flow.sh eval --report # OFFLINE: last complete batch's scorecard + drift
176
+ flow.sh eval --record --n 3 # LIVE: write stripped replay fixtures (billable)
177
+ flow.sh eval --replay --n 3 # KEYLESS: replay recorded transcripts
89
178
  ```
90
179
 
91
180
  Exit code: `0` = every evaluated fixture majority-matched its expected verdict (or a clean
@@ -146,9 +235,10 @@ yet (the same "measure, don't assume" discipline that gated this build in the fi
146
235
  platform: a signal arriving while the runner is blocked on a foreground judge call was not
147
236
  observed to preempt that call promptly. Cleanup between fixtures/calls fires reliably; a
148
237
  normal (uninterrupted) run always cleans up its temp files immediately.
149
- - **Small fixture corpus (seven)**: enough to prove the mechanism and give a real number, not
150
- enough for statistical confidence across every stage/failure-mode combination. Widening the
151
- corpus is a natural v1.x follow-up, not a v1 blocker.
238
+ - **Small fixture corpus (eleven manifest rows, nine heading-mapped)**: enough to prove the
239
+ mechanism and give a real number, not enough for statistical confidence across every
240
+ stage/failure-mode combination. Widening the corpus is a natural v1.x follow-up, not a
241
+ v1 blocker.
152
242
 
153
243
  ## Failure modes and postmortem (v0.21)
154
244
 
@@ -174,8 +264,8 @@ window; the mechanism was never confirmed at the time (rate-limit vs hook conten
174
264
  4. **Circuit breaker.** The FIRST evaluated fixture coming back UNRELIABLE (`invalid_count*3 > n`)
175
265
  aborts the batch with a distinct nonzero exit code (2) and NO `done` trailer written — the
176
266
  junk batch is invisible to `--report`/drift, so it cannot poison the baseline. `--keep-going`
177
- forces the full batch; document worst case (≤ N_fixtures × n × 2 + 1 probe ≈ 37 calls at
178
- default 7 × 3).
267
+ forces the full batch; document worst case (≤ N_fixtures × n × 2 + 1 probe ≈ 55 calls at
268
+ default 9 heading-mapped × 3).
179
269
 
180
270
  ### Rate-limit visibility (advisory, best-effort)
181
271
 
@@ -182,6 +182,9 @@ evidence non-empty.
182
182
  - **Is `## Evidence` real world-state** — a clickable URL, real curl output, a DB row — and
183
183
  NOT "tests pass" / "code merged" / "deployed successfully"? Merge != shipped: the proof is
184
184
  the live surface changing, verified as a user.
185
+ - **Does every `## Evidence` item name its artifact (path/URL) or the command that produced
186
+ it?** Plausible prose that names neither is still hollow even when the mechanical floor
187
+ passes (`ground-truth-gates.md` rule 8).
185
188
  - **If this card fixes a bug/regression**, the evidence includes the red→green proof that the
186
189
  new test was actually tied to the bug (`ground-truth-gates.md` → "Bug-fix cards"), not just a
187
190
  green run.
@@ -40,6 +40,13 @@ and used only to add color, never to pass a gate.
40
40
  7. **Hand-edit `status: done` is not trusted for deps** until Evidence **and** Verify boxes
41
41
  pass the same floor (`ready` re-validates Evidence + unchecked Verify + `[FILL]`; graph
42
42
  executor mirrors).
43
+ 8. **Every done-evidence item names its artifact or command.** Each `## Evidence` item must
44
+ name the path/URL of the artifact or the command that produced it. Plausible prose that
45
+ names neither is hollow even when the mechanical multi-signal floor passes — the card
46
+ gate's semantic challenge catches it (`gate-rules.md` → Card gate). Old prose-only
47
+ evidence still passes `flow.sh check` (the floor is unchanged). **Escalate to full
48
+ structured lineage evidence (B1)** only if hollow-done decoys of this class recur in
49
+ dogfood or live eval after this addendum lands.
43
50
 
44
51
  ## Bug-fix cards: prove the test was tied to the bug
45
52
  When a card's job is fixing a bug/regression (not new behavior), a passing test is not yet
@@ -0,0 +1,54 @@
1
+ # Host-agnostic parallel occupancy
2
+
3
+ Flow requires **no named multiplexer**. Parallel cards are a ready-gate +
4
+ worktrees + host-owned placement + world-state fan-in. Absence of Herdr, tmux,
5
+ in-process `Task`, or any other host **never fails a gate** and never changes
6
+ `flow.sh` exit.
7
+
8
+ Herdr and tmux appear below only as **examples, never as a dependency** or a
9
+ first-class adapter.
10
+
11
+ ## Degrade ladder
12
+
13
+ | Rung | Primitive already in the process | Who binds occupants | Who waits |
14
+ |---|---|---|---|
15
+ | **0** | In-process subagents / `Task` | Hosting agent; cwd = the card's worktree | Host `Task` join |
16
+ | **1** | Operator extra terminals / tabs | Human pastes the print-enter block | Human |
17
+ | **2** | Process is **already inside** a mux (`HERDR_ENV=1` / `TMUX` set) | That mux's **own** skill/CLI; flow does not wrap or detect it | That mux's own wait |
18
+ | **3** | None of the above, or `/flow ready` overlap | Serial: one card | Hosting agent, one at a time |
19
+
20
+ Rung 2 is opt-in by the mux's own environment injection, never by `PATH`.
21
+ Inside-mux probe is the mux's own env (`HERDR_ENV=1`, `TMUX`), never
22
+ `command -v herdr` from outside (a CLI can talk to a same-uid socket without
23
+ `HERDR_ENV`). Absence of rungs 0–2 is the default path, not a degrade footnote.
24
+
25
+ ## Gate parity
26
+
27
+ Host `idle` / `done` / `blocked` is **not a card pass**. `flow.sh check` still
28
+ judges, and it must run **in the card's worktree** (parent-tree check of main
29
+ is a false fail/pass). Screen output is not evidence.
30
+
31
+ Keep the four "blocked" senses distinct:
32
+
33
+ | Sense | Meaning | Home |
34
+ |---|---|---|
35
+ | **host-blocked** | mux/pane approval chrome or occupant stuck | the host |
36
+ | **subagent-BLOCKED** | scoped subagent status protocol | `agent-detection.md` |
37
+ | **ready-blocked** | deps unmet or allowed-files overlap | `flow.sh ready` |
38
+ | **Tier-C security halt** | security-class exposure; operator DEBT | `auto-run.md` |
39
+
40
+ Mixing them auto-answers a payments card or auto-halts a yes/no prompt.
41
+
42
+ ## File-is-the-wait
43
+
44
+ The **HOST waits** (Task join / human / the mux's own wait). `flow.sh` **never waits on or polls** another agent. Fan-in is world-state: re-invoke `check` in
45
+ the card worktree after the host's wait returns.
46
+
47
+ ## Forbidden (ADR-0001; not a nit)
48
+
49
+ `flow.sh` must not exec host control CLIs (`herdr agent start`, `herdr agent wait`, `herdr agent prompt`, `herdr server stop`, `tmux send-keys`, and equivalents).
50
+ Forbidden by name: `mux-up`, `mux-run-wave`, `mux-proxy`. No screen output as
51
+ evidence. No hook install into `~/.claude` or `~/.omp`. No long-lived socket subscribe (`events.subscribe`). `flow.sh` never spawns agents; placement is print-enter or the host's own Task/mux.
52
+
53
+ Own-runtime / own-PTY / own-wait is a successor ADR plus a separate product,
54
+ not a card in this skill.
@@ -69,6 +69,8 @@ HARNESS_PY="$SCRIPT_DIR/../harness/flow_harness.py"
69
69
  # so an overridden manifest can only ever point at real shipped fixtures under the real EVAL_DIR.
70
70
  EVAL_DIR="$SCRIPT_DIR/../eval"
71
71
  EVAL_MANIFEST="${FLOW_EVAL_MANIFEST:-$EVAL_DIR/manifest.tsv}"
72
+ # FLOW_EVAL_REPLAY_DIR is a TEST-ONLY override (synthetic harness-built transcripts).
73
+ EVAL_REPLAY_DIR="${FLOW_EVAL_REPLAY_DIR:-$EVAL_DIR/replay}"
72
74
  GATE_RULES_FILE="$SCRIPT_DIR/../references/gate-rules.md"
73
75
 
74
76
  # Routing eval (v0.22 concierge routing judge - a SEPARATE modality from the artifact-vs-
@@ -2300,6 +2302,14 @@ cmd_doctor() {
2300
2302
  command -v cargo >/dev/null 2>&1 && echo " cargo: ok (optional Rust harness power-path)" || true
2301
2303
  [ -d "$TEMPLATE_DIR" ] && echo " templates: ok" || { echo " templates: MISSING ($TEMPLATE_DIR)"; ok=0; }
2302
2304
  [ -f "$HARNESS_PY" ] && echo " harness: present ($HARNESS_PY)" || echo " harness: absent (durable layer off)"
2305
+ if command -v timeout >/dev/null 2>&1 || command -v gtimeout >/dev/null 2>&1; then
2306
+ echo " timeout: full (timeout|gtimeout present)"
2307
+ else
2308
+ case "$(uname -s 2>/dev/null)" in
2309
+ Darwin) echo " timeout: absent (no timeout/gtimeout; live eval REFUSED)" ;;
2310
+ *) echo " timeout: partial (watchdog fallback; live eval not GNU-bounded)" ;;
2311
+ esac
2312
+ fi
2303
2313
  echo
2304
2314
  if [ "$ok" -eq 1 ]; then
2305
2315
  echo "READY. Gate engine works$([ -n "$py" ] && echo '; durable layer on.' || echo '; durable layer off (no python).')"
@@ -3197,6 +3207,25 @@ _eval_engine_run() { # $1=promptfile $2=timeout-seconds -> stdout=raw json; retu
3197
3207
  # command string above. Empirical: adding a `2> '$errfile'` inside the timed cmd breaks the
3198
3208
  # watchdog-fallback path in `_run_with_timeout` on the timeout-less-PATH lane (macOS DEBT lane).
3199
3209
  # Outer-scope redirection is exactly the same capture at zero cost to the timeout contract.
3210
+ # Opt-in isolated cwd for live eval (Approach D). Unset = today's cwd (byte-identical
3211
+ # call path). Set but not a directory = fail loud. Isolation is cd+env around existing
3212
+ # probe/engine calls — never argv, never a second engine. CLAUDE_CODE_SAFE_MODE=1 is
3213
+ # exported ONLY in the isolated subshell (official Claude env; keeps OAuth; blast radius
3214
+ # is the judge child). Do not invent a flow alias; do not set HOME or CLAUDE_CONFIG_DIR.
3215
+ _eval_isolated_guard() {
3216
+ [ -z "${FLOW_EVAL_ISOLATED_CWD:-}" ] && return 0
3217
+ [ -d "$FLOW_EVAL_ISOLATED_CWD" ] && return 0
3218
+ echo "FAIL: FLOW_EVAL_ISOLATED_CWD is set but is not a directory: $FLOW_EVAL_ISOLATED_CWD"
3219
+ return 1
3220
+ }
3221
+ _eval_isolated_run() {
3222
+ if [ -n "${FLOW_EVAL_ISOLATED_CWD:-}" ]; then
3223
+ ( cd "$FLOW_EVAL_ISOLATED_CWD" && export CLAUDE_CODE_SAFE_MODE=1 && "$@" )
3224
+ else
3225
+ "$@"
3226
+ fi
3227
+ }
3228
+
3200
3229
 
3201
3230
  # Build the judge prompt for one fixture. Returns 1 (writes nothing usable) if the gate-rules.md
3202
3231
  # section extraction came back empty.
@@ -3289,6 +3318,13 @@ _eval_gate_rules_sha() {
3289
3318
  tr -d '\r' < "$GATE_RULES_FILE" | cksum | awk '{print $1}'
3290
3319
  }
3291
3320
 
3321
+ # CRLF-normalized assembler-output hash (same recipe as _eval_gate_rules_sha). Covers
3322
+ # prompt-file bytes only — not host CLAUDE.md. $1=prompt file.
3323
+ _eval_prompt_sha() {
3324
+ [ -f "$1" ] || { printf 'unknown'; return; }
3325
+ tr -d '\r' < "$1" | cksum | awk '{print $1}'
3326
+ }
3327
+
3292
3328
  # `claude --version` is the CLI version (e.g. "2.1.201 (Claude Code)"), NOT a model id - kept
3293
3329
  # as its own axis distinct from `_eval_parse_model` (red-team finding #4: 3 reviewers
3294
3330
  # independently confirmed conflating the two is wrong). `< /dev/null`: this and every other
@@ -3302,6 +3338,48 @@ _eval_cli_version() {
3302
3338
  [ -n "$v" ] && printf '%s' "$v" || printf 'unknown'
3303
3339
  }
3304
3340
 
3341
+ # Replay/record helpers. Do not live inside _eval_engine_run. Meta is key=value, one per line.
3342
+ _eval_replay_meta_get() { # $1=key -> value (empty + return 1 if missing)
3343
+ local key="$1" line
3344
+ [ -f "$EVAL_REPLAY_DIR/meta" ] || return 1
3345
+ while IFS= read -r line || [ -n "$line" ]; do
3346
+ line="${line%$'\r'}"
3347
+ case "$line" in
3348
+ "${key}="*) printf '%s' "${line#${key}=}"; return 0 ;;
3349
+ esac
3350
+ done < "$EVAL_REPLAY_DIR/meta"
3351
+ return 1
3352
+ }
3353
+
3354
+ _eval_replay_write_meta() { # $1=nonce $2=gate_rules_sha $3=model $4=n [$5=prompt_sha.<fid>= lines]
3355
+ mkdir -p "$EVAL_REPLAY_DIR" || return 1
3356
+ printf 'nonce=%s\ngate_rules_sha=%s\nmodel=%s\nn=%s\n' "$1" "$2" "$3" "$4" > "$EVAL_REPLAY_DIR/meta" || return 1
3357
+ [ -n "${5:-}" ] && printf '%s' "$5" >> "$EVAL_REPLAY_DIR/meta"
3358
+ }
3359
+
3360
+ # Persist one stripped verdict line only — never a JSON envelope (session_id/cwd stay out).
3361
+ _eval_fid_safe() { # $1=fid -> filename-safe token (no path separators)
3362
+ local s
3363
+ s="$(printf '%s' "$1" | tr -c 'A-Za-z0-9' '-' | sed -E 's/-+/-/g; s/^-//; s/-$//')"
3364
+ [ -n "$s" ] || s="_"
3365
+ printf '%s' "$s"
3366
+ }
3367
+
3368
+ _eval_record_vote() { # $1=fid $2=vote-i $3=nonce $4=FLAG|PASS|INVALID
3369
+ local dest="$EVAL_REPLAY_DIR/$(_eval_fid_safe "$1")"
3370
+ mkdir -p "$dest" || return 1
3371
+ printf 'GATE-EVAL-%s: %s\n' "$3" "$4" > "$dest/$2.txt"
3372
+ }
3373
+
3374
+ _eval_replay_vote() { # $1=fid $2=vote-i -> transcript on stdout; return 1 if missing
3375
+ local f="$EVAL_REPLAY_DIR/$(_eval_fid_safe "$1")/$2.txt"
3376
+ if [ ! -f "$f" ]; then
3377
+ echo "eval --replay: missing fixture $1 vote $2 at $f" >&2
3378
+ return 1
3379
+ fi
3380
+ cat "$f"
3381
+ }
3382
+
3305
3383
  # Batch header/trailer: the completeness signal `--report`/drift filter on is "start present +
3306
3384
  # done present + n_written >= n_expected". Two paths lead to a MISSING trailer (correctly
3307
3385
  # incomplete): (a) an interrupted batch (INT/TERM trap exits before this line, per the eval
@@ -3323,17 +3401,18 @@ _eval_emit_batch_marker() { # $1=run_id $2=start|done $3=n_expected-or-n_written
3323
3401
  # cli_version/flow_version/gate_rules_sha are passed in (computed ONCE per batch by the caller,
3324
3402
  # not per fixture row) - they never change mid-batch, and recomputing per-row was both wasteful
3325
3403
  # (an extra subprocess per fixture) and the original vector for the stdin-consumption bug above.
3326
- _eval_emit_result() { # $1..$14 as before + $15=retries $16=rate_limited (true|false)
3404
+ _eval_emit_result() { # $1..$14 as before + $15=retries $16=rate_limited $17=prompt_sha $18=timed_out
3327
3405
  local run_id="$1" fid="$2" stage="$3" expected="$4" verdict="$5" match="$6" flag="$7" pass="$8" invalid="$9" n="${10}" model="${11}" cliv="${12}" flowv="${13}" grsha="${14}"
3328
- local retries="${15:-0}" ratel="${16:-false}"
3406
+ local retries="${15:-0}" ratel="${16:-false}" psha="${17:-}" timedout="${18:-false}"
3329
3407
  case "$retries" in ''|*[!0-9]*) retries=0 ;; esac
3330
3408
  case "$ratel" in true|false) : ;; *) ratel=false ;; esac
3409
+ case "$timedout" in true|false) : ;; *) timedout=false ;; esac
3331
3410
  mkdir -p "$LOG_DIR" 2>/dev/null || true
3332
- printf '{"ts":"%s","epoch_s":%s,"run_id":"%s","fixture":"%s","stage":"%s","expected":"%s","verdict":"%s","match":"%s","votes":{"flag":%s,"pass":%s,"invalid":%s},"n":%s,"cli_version":"%s","model":"%s","flow_version":"%s","gate_rules_sha":"%s","retries":%s,"rate_limited":%s}\n' \
3411
+ printf '{"ts":"%s","epoch_s":%s,"run_id":"%s","fixture":"%s","stage":"%s","expected":"%s","verdict":"%s","match":"%s","votes":{"flag":%s,"pass":%s,"invalid":%s},"n":%s,"cli_version":"%s","model":"%s","flow_version":"%s","gate_rules_sha":"%s","retries":%s,"rate_limited":%s,"prompt_sha":"%s","timed_out":%s}\n' \
3333
3412
  "$(date -u +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || echo unknown)" "$(_now)" "$(_json_str "$run_id")" "$(_json_str "$fid")" \
3334
3413
  "$(_json_str "$stage")" "$(_json_str "$expected")" "$(_json_str "$verdict")" "$(_json_str "$match")" \
3335
3414
  "$flag" "$pass" "$invalid" "$n" "$(_json_str "$cliv")" "$(_json_str "$model")" \
3336
- "$(_json_str "$flowv")" "$(_json_str "$grsha")" "$retries" "$ratel" >> "$EVAL_RESULTS_FILE" 2>/dev/null || true
3415
+ "$(_json_str "$flowv")" "$(_json_str "$grsha")" "$retries" "$ratel" "$(_json_str "$psha")" "$timedout" >> "$EVAL_RESULTS_FILE" 2>/dev/null || true
3337
3416
  }
3338
3417
 
3339
3418
  # Strip the noisy `system`/`init` envelope from a raw --output-format json line before persisting
@@ -3617,17 +3696,18 @@ _eval_routing_parse_action() {
3617
3696
  [ -n "$action" ] && printf '%s' "$action" || echo INVALID
3618
3697
  }
3619
3698
 
3620
- _eval_routing_emit_result() { # $1=run_id $2=fid $3=expected $4=verdict $5=match $6..$9=votes-hit/miss/invalid/n $10=model $11=cliv $12=flowv $13=rulessha $14=retries $15=rate_limited
3699
+ _eval_routing_emit_result() { # $1=run_id $2=fid $3=expected $4=verdict $5=match $6..$9=votes $10=model $11=cliv $12=flowv $13=rulessha $14=retries $15=rate_limited $16=prompt_sha $17=timed_out
3621
3700
  local run_id="$1" fid="$2" expected="$3" verdict="$4" match="$5" hit="$6" miss="$7" invalid="$8" n="$9"
3622
- local model="${10}" cliv="${11}" flowv="${12}" rulessha="${13}" retries="${14:-0}" ratel="${15:-false}"
3701
+ local model="${10}" cliv="${11}" flowv="${12}" rulessha="${13}" retries="${14:-0}" ratel="${15:-false}" psha="${16:-}" timedout="${17:-false}"
3623
3702
  case "$retries" in ''|*[!0-9]*) retries=0 ;; esac
3624
3703
  case "$ratel" in true|false) : ;; *) ratel=false ;; esac
3704
+ case "$timedout" in true|false) : ;; *) timedout=false ;; esac
3625
3705
  mkdir -p "$LOG_DIR" 2>/dev/null || true
3626
- printf '{"ts":"%s","epoch_s":%s,"run_id":"%s","fixture":"%s","expected":"%s","verdict":"%s","match":"%s","votes":{"hit":%s,"miss":%s,"invalid":%s},"n":%s,"cli_version":"%s","model":"%s","flow_version":"%s","rules_sha":"%s","retries":%s,"rate_limited":%s}\n' \
3706
+ printf '{"ts":"%s","epoch_s":%s,"run_id":"%s","fixture":"%s","expected":"%s","verdict":"%s","match":"%s","votes":{"hit":%s,"miss":%s,"invalid":%s},"n":%s,"cli_version":"%s","model":"%s","flow_version":"%s","rules_sha":"%s","retries":%s,"rate_limited":%s,"prompt_sha":"%s","timed_out":%s}\n' \
3627
3707
  "$(date -u +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || echo unknown)" "$(_now)" "$(_json_str "$run_id")" "$(_json_str "$fid")" \
3628
3708
  "$(_json_str "$expected")" "$(_json_str "$verdict")" "$(_json_str "$match")" \
3629
3709
  "$hit" "$miss" "$invalid" "$n" "$(_json_str "$cliv")" "$(_json_str "$model")" \
3630
- "$(_json_str "$flowv")" "$(_json_str "$rulessha")" "$retries" "$ratel" >> "$EVAL_ROUTING_RESULTS_FILE" 2>/dev/null || true
3710
+ "$(_json_str "$flowv")" "$(_json_str "$rulessha")" "$retries" "$ratel" "$(_json_str "$psha")" "$timedout" >> "$EVAL_ROUTING_RESULTS_FILE" 2>/dev/null || true
3631
3711
  }
3632
3712
 
3633
3713
  _eval_routing_complete_run_ids() {
@@ -3727,7 +3807,9 @@ function cmd_eval_routing { # same arg shape as cmd_eval, called only when --sta
3727
3807
  return 1
3728
3808
  fi
3729
3809
 
3730
- local probe; probe="$(_eval_probe)"
3810
+ _eval_isolated_guard || return 1
3811
+
3812
+ local probe; probe="$(_eval_isolated_run _eval_probe)"
3731
3813
  case "$probe" in
3732
3814
  absent)
3733
3815
  echo "SKIP: 'claude' CLI not found on PATH - routing eval needs it to run the LLM judge. Zero calls made."
@@ -3795,14 +3877,18 @@ function cmd_eval_routing { # same arg shape as cmd_eval, called only when --sta
3795
3877
  continue
3796
3878
  fi
3797
3879
 
3880
+ local psha
3881
+ psha="$(_eval_prompt_sha "$promptfile")"
3882
+
3798
3883
  total_evaluated=$((total_evaluated + 1))
3799
- local hit_count=0 miss_count=0 invalid_count=0 i=1 model="unknown" retries_sum=0 vote_rate_limited=false
3884
+ local hit_count=0 miss_count=0 invalid_count=0 i=1 model="unknown" retries_sum=0 vote_rate_limited=false vote_timed_out=false
3800
3885
  while [ "$i" -le "$n" ]; do
3801
3886
  local raw rc v raw1="" rc1=0
3802
- raw="$(_eval_engine_run "$promptfile" "$timeout" 2>/dev/null)"; rc=$?
3887
+ raw="$(_eval_isolated_run _eval_engine_run "$promptfile" "$timeout" 2>/dev/null)"; rc=$?
3803
3888
  raw1="$raw"; rc1="$rc"
3804
3889
  if [ "$rc" -eq 124 ]; then
3805
3890
  v=INVALID
3891
+ vote_timed_out=true
3806
3892
  else
3807
3893
  v="$(_eval_routing_parse_action "$raw" "$nonce")"
3808
3894
  [ "$model" = "unknown" ] && model="$(_eval_parse_model "$raw")"
@@ -3814,9 +3900,9 @@ function cmd_eval_routing { # same arg shape as cmd_eval, called only when --sta
3814
3900
  echo " $fid: retrying vote $i after ${backoff}s (parse-INVALID on attempt 1)"
3815
3901
  sleep "$backoff" 2>/dev/null || true
3816
3902
  fi
3817
- raw="$(_eval_engine_run "$promptfile" "$timeout" 2>/dev/null)"; rc=$?
3903
+ raw="$(_eval_isolated_run _eval_engine_run "$promptfile" "$timeout" 2>/dev/null)"; rc=$?
3818
3904
  retries_sum=$((retries_sum + 1))
3819
- if [ "$rc" -eq 124 ]; then v=INVALID; else v="$(_eval_routing_parse_action "$raw" "$nonce")"; fi
3905
+ if [ "$rc" -eq 124 ]; then v=INVALID; vote_timed_out=true; else v="$(_eval_routing_parse_action "$raw" "$nonce")"; fi
3820
3906
  [ "$model" = "unknown" ] && [ "$rc" -ne 124 ] && model="$(_eval_parse_model "$raw")"
3821
3907
  local rl2; rl2="$(_eval_parse_rate_limited "$raw")"
3822
3908
  [ "$rl2" = "true" ] && vote_rate_limited=true
@@ -3854,7 +3940,7 @@ function cmd_eval_routing { # same arg shape as cmd_eval, called only when --sta
3854
3940
  echo " $fid: MISS - expected '$fexpected' (hit=$hit_count miss=$miss_count invalid=$invalid_count)"
3855
3941
  fi
3856
3942
 
3857
- _eval_routing_emit_result "$nonce" "$fid" "$fexpected" "$verdict" "$match" "$hit_count" "$miss_count" "$invalid_count" "$n" "$model" "$batch_cliv" "$batch_flowv" "$batch_rulessha" "$retries_sum" "$vote_rate_limited"
3943
+ _eval_routing_emit_result "$nonce" "$fid" "$fexpected" "$verdict" "$match" "$hit_count" "$miss_count" "$invalid_count" "$n" "$model" "$batch_cliv" "$batch_flowv" "$batch_rulessha" "$retries_sum" "$vote_rate_limited" "$psha" "$vote_timed_out"
3858
3944
  n_written=$((n_written + 1))
3859
3945
 
3860
3946
  if [ "$total_evaluated" -eq 1 ] && [ "$verdict" = "UNRELIABLE" ] && [ "$keep_going" -eq 0 ]; then
@@ -3962,15 +4048,16 @@ _eval_converge_parse_verdict() { # $1=raw $2=nonce -> GAP|CONVERGED|INVALID
3962
4048
 
3963
4049
  _eval_converge_emit_result() { # same field shape as _eval_routing_emit_result
3964
4050
  local run_id="$1" fid="$2" expected="$3" verdict="$4" match="$5" hit="$6" miss="$7" invalid="$8" n="$9"
3965
- local model="${10}" cliv="${11}" flowv="${12}" rulessha="${13}" retries="${14:-0}" ratel="${15:-false}"
4051
+ local model="${10}" cliv="${11}" flowv="${12}" rulessha="${13}" retries="${14:-0}" ratel="${15:-false}" psha="${16:-}" timedout="${17:-false}"
3966
4052
  case "$retries" in ''|*[!0-9]*) retries=0 ;; esac
3967
4053
  case "$ratel" in true|false) : ;; *) ratel=false ;; esac
4054
+ case "$timedout" in true|false) : ;; *) timedout=false ;; esac
3968
4055
  mkdir -p "$LOG_DIR" 2>/dev/null || true
3969
- printf '{"ts":"%s","epoch_s":%s,"run_id":"%s","fixture":"%s","expected":"%s","verdict":"%s","match":"%s","votes":{"hit":%s,"miss":%s,"invalid":%s},"n":%s,"cli_version":"%s","model":"%s","flow_version":"%s","rules_sha":"%s","retries":%s,"rate_limited":%s}\n' \
4056
+ printf '{"ts":"%s","epoch_s":%s,"run_id":"%s","fixture":"%s","expected":"%s","verdict":"%s","match":"%s","votes":{"hit":%s,"miss":%s,"invalid":%s},"n":%s,"cli_version":"%s","model":"%s","flow_version":"%s","rules_sha":"%s","retries":%s,"rate_limited":%s,"prompt_sha":"%s","timed_out":%s}\n' \
3970
4057
  "$(date -u +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || echo unknown)" "$(_now)" "$(_json_str "$run_id")" "$(_json_str "$fid")" \
3971
4058
  "$(_json_str "$expected")" "$(_json_str "$verdict")" "$(_json_str "$match")" \
3972
4059
  "$hit" "$miss" "$invalid" "$n" "$(_json_str "$cliv")" "$(_json_str "$model")" \
3973
- "$(_json_str "$flowv")" "$(_json_str "$rulessha")" "$retries" "$ratel" >> "$EVAL_CONVERGE_RESULTS_FILE" 2>/dev/null || true
4060
+ "$(_json_str "$flowv")" "$(_json_str "$rulessha")" "$retries" "$ratel" "$(_json_str "$psha")" "$timedout" >> "$EVAL_CONVERGE_RESULTS_FILE" 2>/dev/null || true
3974
4061
  }
3975
4062
 
3976
4063
  _eval_converge_emit_batch_marker() { # $1=run_id $2=start|done $3=n
@@ -4032,7 +4119,9 @@ function cmd_eval_converge { # same arg shape as cmd_eval, called only when --st
4032
4119
  return 1
4033
4120
  fi
4034
4121
 
4035
- local probe; probe="$(_eval_probe)"
4122
+ _eval_isolated_guard || return 1
4123
+
4124
+ local probe; probe="$(_eval_isolated_run _eval_probe)"
4036
4125
  case "$probe" in
4037
4126
  absent) echo "SKIP: 'claude' CLI not found on PATH - converge eval needs it to run the LLM judge. Zero calls made."; return 0 ;;
4038
4127
  fail) echo "SKIP: 'claude' CLI present but the sentinel probe did not come back clean (one minimal probe call was made). Retry once 'claude -p' runs headless."; return 0 ;;
@@ -4080,17 +4169,20 @@ function cmd_eval_converge { # same arg shape as cmd_eval, called only when --st
4080
4169
  echo " $fid: FAIL - could not build converge prompt (converge.md/repo-dir/challenge missing)"; total_miss=$((total_miss + 1)); rm -rf "$rundir" 2>/dev/null; continue
4081
4170
  fi
4082
4171
 
4172
+ local psha
4173
+ psha="$(_eval_prompt_sha "$promptfile")"
4174
+
4083
4175
  total_evaluated=$((total_evaluated + 1))
4084
- local hit_count=0 miss_count=0 invalid_count=0 i=1 model="unknown" retries_sum=0 vote_rate_limited=false
4176
+ local hit_count=0 miss_count=0 invalid_count=0 i=1 model="unknown" retries_sum=0 vote_rate_limited=false vote_timed_out=false
4085
4177
  while [ "$i" -le "$n" ]; do
4086
4178
  local raw rc v raw1="" rc1=0
4087
- raw="$(_eval_engine_run "$promptfile" "$timeout" 2>/dev/null)"; rc=$?; raw1="$raw"; rc1="$rc"
4088
- if [ "$rc" -eq 124 ]; then v=INVALID; else v="$(_eval_converge_parse_verdict "$raw" "$nonce")"; [ "$model" = "unknown" ] && model="$(_eval_parse_model "$raw")"; fi
4179
+ raw="$(_eval_isolated_run _eval_engine_run "$promptfile" "$timeout" 2>/dev/null)"; rc=$?; raw1="$raw"; rc1="$rc"
4180
+ if [ "$rc" -eq 124 ]; then v=INVALID; vote_timed_out=true; else v="$(_eval_converge_parse_verdict "$raw" "$nonce")"; [ "$model" = "unknown" ] && model="$(_eval_parse_model "$raw")"; fi
4089
4181
  local rl1; rl1="$(_eval_parse_rate_limited "$raw1")"; [ "$rl1" = "true" ] && vote_rate_limited=true
4090
4182
  if [ "$v" = "INVALID" ] && [ "$rl1" != "true" ] && [ "$rc1" -ne 124 ]; then
4091
4183
  if [ "$backoff" -gt 0 ]; then echo " $fid: retrying vote $i after ${backoff}s (parse-INVALID on attempt 1)"; sleep "$backoff" 2>/dev/null || true; fi
4092
- raw="$(_eval_engine_run "$promptfile" "$timeout" 2>/dev/null)"; rc=$?; retries_sum=$((retries_sum + 1))
4093
- if [ "$rc" -eq 124 ]; then v=INVALID; else v="$(_eval_converge_parse_verdict "$raw" "$nonce")"; fi
4184
+ raw="$(_eval_isolated_run _eval_engine_run "$promptfile" "$timeout" 2>/dev/null)"; rc=$?; retries_sum=$((retries_sum + 1))
4185
+ if [ "$rc" -eq 124 ]; then v=INVALID; vote_timed_out=true; else v="$(_eval_converge_parse_verdict "$raw" "$nonce")"; fi
4094
4186
  [ "$model" = "unknown" ] && [ "$rc" -ne 124 ] && model="$(_eval_parse_model "$raw")"
4095
4187
  local rl2; rl2="$(_eval_parse_rate_limited "$raw")"; [ "$rl2" = "true" ] && vote_rate_limited=true
4096
4188
  fi
@@ -4115,7 +4207,7 @@ function cmd_eval_converge { # same arg shape as cmd_eval, called only when --st
4115
4207
  total_miss=$((total_miss + 1)); echo " $fid: MISS - expected '$fexpected' (hit=$hit_count miss=$miss_count invalid=$invalid_count)"
4116
4208
  fi
4117
4209
 
4118
- _eval_converge_emit_result "$nonce" "$fid" "$fexpected" "$verdict" "$match" "$hit_count" "$miss_count" "$invalid_count" "$n" "$model" "$batch_cliv" "$batch_flowv" "$batch_rulessha" "$retries_sum" "$vote_rate_limited"
4210
+ _eval_converge_emit_result "$nonce" "$fid" "$fexpected" "$verdict" "$match" "$hit_count" "$miss_count" "$invalid_count" "$n" "$model" "$batch_cliv" "$batch_flowv" "$batch_rulessha" "$retries_sum" "$vote_rate_limited" "$psha" "$vote_timed_out"
4119
4211
  n_written=$((n_written + 1))
4120
4212
 
4121
4213
  if [ "$total_evaluated" -eq 1 ] && [ "$verdict" = "UNRELIABLE" ] && [ "$keep_going" -eq 0 ]; then
@@ -4138,6 +4230,30 @@ function cmd_eval_converge { # same arg shape as cmd_eval, called only when --st
4138
4230
  return 1
4139
4231
  }
4140
4232
 
4233
+ # Live-eval entry guard (macOS DEBT). Refuse artifact eval on darwin when no real
4234
+ # timeout/gtimeout can bound a stuck claude call. Replay (Phase 7 sets replay_mode=1)
4235
+ # never hits this. --report returns before the call site. Do not move this into
4236
+ # _eval_engine_run or _run_with_timeout.
4237
+ _eval_guard_unbounded_darwin() { # $1=replay_mode (0/1); return 1 = refused
4238
+ local replay_mode="${1:-0}"
4239
+ [ "$replay_mode" -eq 0 ] || return 0
4240
+ [ "${FLOW_EVAL_UNBOUNDED:-}" = "1" ] && return 0
4241
+ local darwin=0
4242
+ if [ "${FLOW_EVAL_FORCE_DARWIN:-}" = "1" ]; then
4243
+ darwin=1
4244
+ else
4245
+ case "$(uname -s 2>/dev/null)" in Darwin) darwin=1 ;; esac
4246
+ fi
4247
+ [ "$darwin" -eq 1 ] || return 0
4248
+ if command -v timeout >/dev/null 2>&1 || command -v gtimeout >/dev/null 2>&1; then
4249
+ return 0
4250
+ fi
4251
+ echo "REFUSED: live eval on macOS without timeout/gtimeout cannot bound a stuck claude call."
4252
+ echo " Unbounded-billing risk: a hung judge can run past --timeout (DEBT.md)."
4253
+ echo " Install GNU coreutils (gtimeout) or export FLOW_EVAL_UNBOUNDED=1 to accept that risk."
4254
+ return 1
4255
+ }
4256
+
4141
4257
  # NOTE: defined with the `function name { }` form (no parens) rather than this file's usual
4142
4258
  # `name() { }` style, solely so the literal 4-byte substring the shipped verb name is built
4143
4259
  # from never appears immediately followed by '(' in the source - a blind text-pattern security
@@ -4145,7 +4261,7 @@ function cmd_eval_converge { # same arg shape as cmd_eval, called only when --st
4145
4261
  # -positives on that exact byte sequence regardless of language. Purely a spelling dodge for a
4146
4262
  # generic scanner; behavior is 100% identical to every other function in this file.
4147
4263
  function cmd_eval {
4148
- local stage_filter="" fixture_filter="" n=3 timeout=120 report_mode=0 keep_going=0
4264
+ local stage_filter="" fixture_filter="" n=3 timeout=120 report_mode=0 keep_going=0 replay_mode=0 record_mode=0
4149
4265
  while [ $# -gt 0 ]; do
4150
4266
  case "$1" in
4151
4267
  --stage) shift; stage_filter="${1:-}" ;;
@@ -4153,8 +4269,10 @@ function cmd_eval {
4153
4269
  --n) shift; n="${1:-3}" ;;
4154
4270
  --timeout) shift; timeout="${1:-120}" ;;
4155
4271
  --report) report_mode=1 ;;
4272
+ --record) record_mode=1 ;;
4273
+ --replay) replay_mode=1 ;;
4156
4274
  --keep-going) keep_going=1 ;;
4157
- *) echo "usage: /flow eval [--stage 01|02|05|card|routing|converge] [--fixture <id>] [--n 3] [--timeout <seconds>] [--keep-going] [--report]"; return 1 ;;
4275
+ *) echo "usage: /flow eval [--stage 01|02|05|card|routing|converge] [--fixture <id>] [--n 3] [--timeout <seconds>] [--keep-going] [--report] [--record|--replay]"; return 1 ;;
4158
4276
  esac
4159
4277
  shift 2>/dev/null || true
4160
4278
  done
@@ -4167,6 +4285,26 @@ function cmd_eval {
4167
4285
  local backoff="${FLOW_EVAL_RETRY_BACKOFF:-5}"
4168
4286
  case "$backoff" in ''|*[!0-9]*) backoff=5 ;; esac
4169
4287
 
4288
+ # Reject window: --record|--replay is artifact-modality only. Must run BEFORE routing/
4289
+ # converge dispatch and BEFORE the --report early-return. --record is LIVE (replay_mode
4290
+ # stays 0) so the Phase 6 darwin guard still fires.
4291
+ if [ "$record_mode" -eq 1 ] && [ "$replay_mode" -eq 1 ]; then
4292
+ echo "usage: /flow eval --record and --replay are mutually exclusive"
4293
+ return 1
4294
+ fi
4295
+ if [ "$record_mode" -eq 1 ] || [ "$replay_mode" -eq 1 ]; then
4296
+ case "$stage_filter" in
4297
+ routing|converge)
4298
+ echo "usage: /flow eval --record|--replay is artifact-modality only (not --stage $stage_filter)"
4299
+ return 1
4300
+ ;;
4301
+ esac
4302
+ if [ "$report_mode" -eq 1 ]; then
4303
+ echo "usage: /flow eval --record|--replay cannot combine with --report"
4304
+ return 1
4305
+ fi
4306
+ fi
4307
+
4170
4308
  # v0.22: routing is a SEPARATE judge modality (own manifest/prompt/verdict/results-file/
4171
4309
  # scorecard) - delegate entirely and return, before any of the artifact-eval logic below
4172
4310
  # (which assumes gate-rules.md stage sections and FLAG/PASS verdicts) ever runs.
@@ -4200,22 +4338,56 @@ function cmd_eval {
4200
4338
  return 1
4201
4339
  fi
4202
4340
 
4203
- local probe; probe="$(_eval_probe)"
4204
- case "$probe" in
4205
- absent)
4206
- echo "SKIP: 'claude' CLI not found on PATH - eval needs it to run the LLM judge. Zero calls made."
4207
- return 0 ;;
4208
- fail)
4209
- echo "SKIP: 'claude' CLI is present but the sentinel probe did not come back clean (one minimal"
4210
- echo " billable probe call was made). Confirm 'claude -p' runs headless on this machine, then retry."
4211
- return 0 ;;
4212
- esac
4341
+ # Live-eval entry. replay_mode=1 self-skips inside the guard; --record leaves
4342
+ # replay_mode=0 so refuse-by-default still fires. Do not wrap this call again.
4343
+ if ! _eval_guard_unbounded_darwin "$replay_mode"; then
4344
+ return 1
4345
+ fi
4213
4346
 
4214
- local nonce; nonce="$(_eval_nonce)"
4215
- local total_mismatch=0 total_unreliable=0 total_evaluated=0 rundir="" n_written=0 aborted=0
4347
+ local nonce="" total_mismatch=0 total_unreliable=0 total_evaluated=0 rundir="" n_written=0 aborted=0
4216
4348
  # Computed ONCE per batch, not per fixture row - see _eval_emit_result's comment.
4217
- local batch_cliv batch_flowv batch_grsha
4218
- batch_cliv="$(_eval_cli_version)"; batch_flowv="$(_flow_version)"; batch_grsha="$(_eval_gate_rules_sha)"
4349
+ local batch_cliv="" batch_flowv="" batch_grsha="" prompt_sha_acc="" prompt_sha_enforced=0
4350
+ if [ "$replay_mode" -eq 1 ]; then
4351
+ # Replay wrap: skip probe SKIP-exit and _eval_cli_version only. Still compute
4352
+ # batch_grsha (staleness hard-fail) and _flow_version. Nonce from fixture, never _eval_nonce.
4353
+ if [ ! -f "$EVAL_REPLAY_DIR/meta" ]; then
4354
+ echo "eval --replay: missing replay fixtures at $EVAL_REPLAY_DIR (no meta). Never SKIP."
4355
+ return 1
4356
+ fi
4357
+ nonce="$(_eval_replay_meta_get nonce)" || nonce=""
4358
+ if [ -z "$nonce" ]; then
4359
+ echo "eval --replay: meta missing nonce. Never SKIP."
4360
+ return 1
4361
+ fi
4362
+ batch_grsha="$(_eval_gate_rules_sha)"
4363
+ batch_flowv="$(_flow_version)"
4364
+ batch_cliv="replay"
4365
+ local rec_sha
4366
+ rec_sha="$(_eval_replay_meta_get gate_rules_sha)" || rec_sha=""
4367
+ if [ "$rec_sha" != "$batch_grsha" ]; then
4368
+ echo "eval --replay: fixtures stale — gate-rules changed; re-record live per ADR re-baseline rule"
4369
+ echo " recorded=$rec_sha current=$batch_grsha"
4370
+ return 1
4371
+ fi
4372
+ if grep -q '^prompt_sha\.' "$EVAL_REPLAY_DIR/meta" 2>/dev/null; then
4373
+ prompt_sha_enforced=1
4374
+ fi
4375
+ echo "replay: keyless artifact batch (nonce from fixture, zero live calls)"
4376
+ else
4377
+ _eval_isolated_guard || return 1
4378
+ local probe; probe="$(_eval_isolated_run _eval_probe)"
4379
+ case "$probe" in
4380
+ absent)
4381
+ echo "SKIP: 'claude' CLI not found on PATH - eval needs it to run the LLM judge. Zero calls made."
4382
+ return 0 ;;
4383
+ fail)
4384
+ echo "SKIP: 'claude' CLI is present but the sentinel probe did not come back clean (one minimal"
4385
+ echo " billable probe call was made). Confirm 'claude -p' runs headless on this machine, then retry."
4386
+ return 0 ;;
4387
+ esac
4388
+ nonce="$(_eval_nonce)"
4389
+ batch_cliv="$(_eval_cli_version)"; batch_flowv="$(_flow_version)"; batch_grsha="$(_eval_gate_rules_sha)"
4390
+ fi
4219
4391
  echo "eval: batch $nonce (N=$n per fixture, timeout=${timeout}s)"
4220
4392
 
4221
4393
  # Ensure .flow/ (which now hosts eval-raw/ postmortem dumps) is git-ignored on any project
@@ -4246,7 +4418,9 @@ function cmd_eval {
4246
4418
  }
4247
4419
  END { print c + 0 }
4248
4420
  ' "$EVAL_MANIFEST")"
4249
- _eval_emit_batch_marker "$nonce" start "$n_expected"
4421
+ if [ "$replay_mode" -eq 0 ]; then
4422
+ _eval_emit_batch_marker "$nonce" start "$n_expected"
4423
+ fi
4250
4424
 
4251
4425
  # Scoped INT/TERM trap: the sole EXIT trap (_log_on_exit -> _cleanup_tds) was found NOT to
4252
4426
  # fire reliably while blocked on a foreground `_run_with_timeout` child (verified: a signal
@@ -4293,29 +4467,63 @@ function cmd_eval {
4293
4467
 
4294
4468
  local promptfile="$rundir/prompt.txt"
4295
4469
  if ! _eval_build_prompt "$promptfile" "$fstage_short" "$artifact_path" "$nonce"; then
4470
+ if [ "$replay_mode" -eq 1 ]; then
4471
+ echo " $fid: SKIP - no heading-mapped section (artifact-replay scope is 01|02|card)"
4472
+ rm -rf "$rundir" 2>/dev/null
4473
+ continue
4474
+ fi
4296
4475
  echo " $fid: FAIL - gate-rules.md section for stage '$fstage' extracted EMPTY (heading map or file drift) - aborted before any billable call"
4297
4476
  total_mismatch=$((total_mismatch + 1))
4298
4477
  rm -rf "$rundir" 2>/dev/null
4299
4478
  continue
4300
4479
  fi
4301
4480
 
4481
+ local psha
4482
+ psha="$(_eval_prompt_sha "$promptfile")"
4483
+ if [ "$replay_mode" -eq 1 ] && [ "$prompt_sha_enforced" -eq 1 ]; then
4484
+ local rec_psha
4485
+ rec_psha="$(_eval_replay_meta_get "prompt_sha.$fid")" || rec_psha=""
4486
+ if [ -z "$rec_psha" ] || [ "$rec_psha" != "$psha" ]; then
4487
+ echo "eval --replay: fixtures stale — prompt assembly changed; re-record live per ADR re-baseline rule"
4488
+ echo " recorded=$rec_psha current=$psha fixture=$fid"
4489
+ rm -rf "$rundir" 2>/dev/null
4490
+ trap - INT TERM
4491
+ return 1
4492
+ fi
4493
+ fi
4494
+ if [ "$record_mode" -eq 1 ]; then
4495
+ prompt_sha_acc="${prompt_sha_acc}prompt_sha.${fid}=${psha}"$'\n'
4496
+ fi
4497
+
4302
4498
  total_evaluated=$((total_evaluated + 1))
4303
- local flag_count=0 pass_count=0 invalid_count=0 i=1 model="unknown" retries_sum=0 vote_rate_limited=false
4499
+ local flag_count=0 pass_count=0 invalid_count=0 i=1 model="unknown" retries_sum=0 vote_rate_limited=false vote_timed_out=false
4304
4500
  # Filename-safe fixture id for raw-capture paths - manifest fid is only tr -d '\r'-cleaned
4305
4501
  # (the v1 trust-boundary is READ-side only, but we are now WRITING keyed by it), so a
4306
4502
  # hand-edited or FLOW_EVAL_MANIFEST-overridden manifest cannot traverse out of eval-raw/.
4307
4503
  local fid_safe; fid_safe="$(printf '%s' "$fid" | tr -c 'A-Za-z0-9' '-' | sed -E 's/-+/-/g; s/^-//; s/-$//')"
4308
4504
  [ -z "$fid_safe" ] && fid_safe="_"
4505
+ if [ "$replay_mode" -eq 0 ]; then
4506
+ local rawdir_prompt="$LOG_DIR/eval-raw/$nonce"
4507
+ if mkdir -p "$rawdir_prompt" 2>/dev/null; then
4508
+ cp "$promptfile" "$rawdir_prompt/${fid_safe}.prompt" 2>/dev/null || true
4509
+ fi
4510
+ fi
4309
4511
  while [ "$i" -le "$n" ]; do
4310
4512
  local raw rc v raw1="" rc1=0 err1="" err2=""
4311
4513
  # Attempt 1 - stderr captured at the OUTER redirection so it never enters the sh -c
4312
4514
  # command string _eval_engine_run passes to _run_with_timeout (that broke the watchdog
4313
4515
  # fallback on the timeout-less-PATH lane - see engine-run's note).
4314
4516
  err1="$rundir/v${i}-a1.err"
4315
- raw="$(_eval_engine_run "$promptfile" "$timeout" 2>"$err1")"; rc=$?
4517
+ if [ "$replay_mode" -eq 1 ]; then
4518
+ raw="$(_eval_replay_vote "$fid" "$i")" || { rm -rf "$rundir" 2>/dev/null; trap - INT TERM; return 1; }
4519
+ rc=0
4520
+ else
4521
+ raw="$(_eval_isolated_run _eval_engine_run "$promptfile" "$timeout" 2>"$err1")"; rc=$?
4522
+ fi
4316
4523
  raw1="$raw"; rc1="$rc"
4317
4524
  if [ "$rc" -eq 124 ]; then
4318
4525
  v=INVALID
4526
+ vote_timed_out=true
4319
4527
  else
4320
4528
  v="$(_eval_parse_verdict "$raw" "$nonce")"
4321
4529
  [ "$model" = "unknown" ] && model="$(_eval_parse_model "$raw")"
@@ -4332,7 +4540,7 @@ function cmd_eval {
4332
4540
  # _run_with_timeout is documented not to bound a stuck call (DEBT.md), a retry here just
4333
4541
  # waits another full stuck-call duration - test E on the 3-OS CI matrix reproduces this at
4334
4542
  # ~66s (2x ~30s stuck calls + backoff) despite --timeout 3.
4335
- if [ "$v" = "INVALID" ] && [ "$rl1" != "true" ] && [ "$rc1" -ne 124 ]; then
4543
+ if [ "$replay_mode" -eq 0 ] && [ "$v" = "INVALID" ] && [ "$rl1" != "true" ] && [ "$rc1" -ne 124 ]; then
4336
4544
  # Backoff then retry - env-injectable so tests can zero it; skipped when rc=124 (timeout
4337
4545
  # was infra, retry would timeout again) or when rate-limited (see above).
4338
4546
  if [ "$backoff" -gt 0 ]; then
@@ -4342,9 +4550,14 @@ function cmd_eval {
4342
4550
  echo " $fid: retrying vote $i (parse-INVALID on attempt 1)"
4343
4551
  fi
4344
4552
  err2="$rundir/v${i}-a2.err"
4345
- raw="$(_eval_engine_run "$promptfile" "$timeout" 2>"$err2")"; rc=$?
4553
+ if [ "$replay_mode" -eq 1 ]; then
4554
+ raw="$(_eval_replay_vote "$fid" "$i")" || { rm -rf "$rundir" 2>/dev/null; trap - INT TERM; return 1; }
4555
+ rc=0
4556
+ else
4557
+ raw="$(_eval_isolated_run _eval_engine_run "$promptfile" "$timeout" 2>"$err2")"; rc=$?
4558
+ fi
4346
4559
  retries_sum=$((retries_sum + 1))
4347
- if [ "$rc" -eq 124 ]; then v=INVALID; else v="$(_eval_parse_verdict "$raw" "$nonce")"; fi
4560
+ if [ "$rc" -eq 124 ]; then v=INVALID; vote_timed_out=true; else v="$(_eval_parse_verdict "$raw" "$nonce")"; fi
4348
4561
  [ "$model" = "unknown" ] && [ "$rc" -ne 124 ] && model="$(_eval_parse_model "$raw")"
4349
4562
  local rl2; rl2="$(_eval_parse_rate_limited "$raw")"
4350
4563
  [ "$rl2" = "true" ] && vote_rate_limited=true
@@ -4385,6 +4598,9 @@ function cmd_eval {
4385
4598
  PASS) pass_count=$((pass_count + 1)) ;;
4386
4599
  *) invalid_count=$((invalid_count + 1)) ;;
4387
4600
  esac
4601
+ if [ "$record_mode" -eq 1 ]; then
4602
+ _eval_record_vote "$fid" "$i" "$nonce" "$v"
4603
+ fi
4388
4604
  i=$((i + 1))
4389
4605
  done
4390
4606
  rm -rf "$rundir" 2>/dev/null
@@ -4415,7 +4631,9 @@ function cmd_eval {
4415
4631
  echo " $fid ($fstage): $verdict - MISMATCH, expected $fexpected (flag=$flag_count pass=$pass_count invalid=$invalid_count)"
4416
4632
  fi
4417
4633
 
4418
- _eval_emit_result "$nonce" "$fid" "$fstage" "$fexpected" "$verdict" "$match" "$flag_count" "$pass_count" "$invalid_count" "$n" "$model" "$batch_cliv" "$batch_flowv" "$batch_grsha" "$retries_sum" "$vote_rate_limited"
4634
+ if [ "$replay_mode" -eq 0 ]; then
4635
+ _eval_emit_result "$nonce" "$fid" "$fstage" "$fexpected" "$verdict" "$match" "$flag_count" "$pass_count" "$invalid_count" "$n" "$model" "$batch_cliv" "$batch_flowv" "$batch_grsha" "$retries_sum" "$vote_rate_limited" "$psha" "$vote_timed_out"
4636
+ fi
4419
4637
  n_written=$((n_written + 1))
4420
4638
 
4421
4639
  # Circuit breaker: catches the 260710-class INVALID storm early. Trip when the FIRST
@@ -4442,9 +4660,12 @@ function cmd_eval {
4442
4660
  # SKIPS the trailer so an early-broken batch never satisfies --report's completeness check
4443
4661
  # (n_written>=n_expected AND trailer present), even on a --fixture-filtered run where
4444
4662
  # n_written could accidentally equal the filtered n_expected of 1.
4445
- if [ "$aborted" -eq 0 ]; then
4663
+ if [ "$aborted" -eq 0 ] && [ "$replay_mode" -eq 0 ]; then
4446
4664
  _eval_emit_batch_marker "$nonce" done "$n_written"
4447
4665
  fi
4666
+ if [ "$record_mode" -eq 1 ] && [ "$aborted" -eq 0 ]; then
4667
+ _eval_replay_write_meta "$nonce" "$batch_grsha" "${model:-unknown}" "$n" "$prompt_sha_acc"
4668
+ fi
4448
4669
 
4449
4670
  echo
4450
4671
  if [ "$total_evaluated" -eq 0 ]; then
@@ -4457,8 +4678,13 @@ function cmd_eval {
4457
4678
  echo "ABORTED: circuit breaker tripped on first fixture UNRELIABLE - $total_evaluated of $n_expected fixtures evaluated."
4458
4679
  return 2
4459
4680
  fi
4460
- _eval_print_scorecard "$nonce"
4681
+ if [ "$replay_mode" -eq 0 ]; then
4682
+ _eval_print_scorecard "$nonce"
4683
+ fi
4461
4684
  echo
4685
+ if [ "$replay_mode" -eq 1 ]; then
4686
+ echo "replay: $total_evaluated fixture(s) (keyless)"
4687
+ fi
4462
4688
  if [ "$total_mismatch" -eq 0 ] && [ "$total_unreliable" -eq 0 ]; then
4463
4689
  echo "PASS: all $total_evaluated evaluated fixture(s) majority-matched their expected verdict."
4464
4690
  return 0
@@ -4506,9 +4732,14 @@ usage: bash flow.sh <command> [args]
4506
4732
  promote <file> Copy a playbook into the cross-project KB (~/.claude/flow/playbooks)
4507
4733
  doctor Check the environment (bash/python/grep/git) across macOS/Linux/Windows
4508
4734
  eval [opts] Behavioral eval: does the LLM semantic gate flag a hollow-but-mechanically-clean
4509
- fixture? Opt-in, BILLABLE (skips cleanly with zero calls if 'claude' CLI absent).
4735
+ fixture? Opt-in, BILLABLE (live mode skips cleanly with zero calls if 'claude'
4736
+ CLI absent; --replay never inherits that SKIP).
4510
4737
  [--stage 01|02|05|card|routing|converge] [--fixture <id>] [--n 3] [--timeout <seconds>]
4511
4738
  [--report] offline: last complete batch's scorecard + drift, zero calls
4739
+ [--record] live: write stripped transcripts under eval/replay/ (billable)
4740
+ [--replay] keyless: feed recorded transcripts through parse/vote/scorecard
4741
+ --record|--replay are artifact-modality only (not --stage routing|converge,
4742
+ not --report). --replay is not a fresh-judge and never counts toward the floor.
4512
4743
  --stage routing: SEPARATE judge (concierge routing, not gate-rules FLAG/PASS);
4513
4744
  hard-capped at 90 calls/batch, own results stream, own --report
4514
4745
  retro Print the 3 retro questions
@@ -1,8 +1,8 @@
1
1
  {
2
2
  "source": "../skills/flow",
3
3
  "destRelative": "skills/flow",
4
- "fileCount": 100,
5
- "syncedAt": "2026-08-13T12:49:42.684Z",
4
+ "fileCount": 103,
5
+ "syncedAt": "2026-08-20T13:35:30.768Z",
6
6
  "fileList": [
7
7
  "SKILL.md",
8
8
  "_templates/00-idea.md",
@@ -35,6 +35,8 @@
35
35
  "eval/fixtures/fcdb/cards/C-001.md",
36
36
  "eval/fixtures/fcdc/.flow/.gitkeep",
37
37
  "eval/fixtures/fcdc/cards/C-001.md",
38
+ "eval/fixtures/fcdd/cards/C-001.md",
39
+ "eval/fixtures/fcde/cards/C-001.md",
38
40
  "eval/fixtures/routing/manifest.tsv",
39
41
  "eval/fixtures/routing/states/state-any-midproject.txt",
40
42
  "eval/fixtures/routing/states/state-cards-todo.txt",
@@ -95,6 +97,7 @@
95
97
  "references/gate-eval.md",
96
98
  "references/gate-rules.md",
97
99
  "references/ground-truth-gates.md",
100
+ "references/host-agnostic-parallel.md",
98
101
  "references/loop-harness-2026-principles.md",
99
102
  "references/mode-work.md",
100
103
  "references/native-rituals.md",