@manhquy/flow-skill 0.6.0 → 0.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -2,6 +2,29 @@
2
2
 
3
3
  All notable changes to `@manhquy/flow-skill`. Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Versioning: [SemVer](https://semver.org/).
4
4
 
5
+ ## [0.7.0] — 2026-08-14 — ships skill v0.30.0 (discipline-layer identity + CI/eval hardening)
6
+
7
+ Skill-content release. Ships skill product **v0.30.0**: identity ADR (gates and
8
+ receipts, never the runtime), keyless `eval --record`/`--replay`, pack-rehearsal,
9
+ `all-checks-passed`, macOS live-eval refuse-guard, B1-S `fcdd`/`fcde` (see root
10
+ `CHANGELOG.md` `## 0.30.0`). Installer CLI contract unchanged. Publishes to
11
+ dist-tag **`latest`**.
12
+
13
+ ### Changed
14
+ - Bundled skill tree re-synced from monorepo `skills/flow/`.
15
+ - Install: **`npx @manhquy/flow-skill@latest`**.
16
+
17
+ ## [0.6.0] — 2026-08-13 — ships skill v0.29.0 (spec-kit imports + converge)
18
+
19
+ Skill-content release. Ships skill product **v0.29.0**: `/flow clarify`, card
20
+ independent-test field, `/flow converge`, converge eval modality (see root
21
+ `CHANGELOG.md` `## 0.29.0`). Installer CLI contract unchanged. Publishes to
22
+ dist-tag **`latest`**.
23
+
24
+ ### Changed
25
+ - Bundled skill tree re-synced from monorepo `skills/flow/`.
26
+ - Install: **`npx @manhquy/flow-skill@latest`**.
27
+
5
28
  ## [0.5.0] — 2026-08-11 — ships skill v0.28.0 (attested execution)
6
29
 
7
30
  Installer minor: bundles skill **v0.28.0** (risk + attestation receipts + auto trust plane).
package/README.md CHANGED
@@ -5,10 +5,11 @@ Install the **flow** skill into coding agents with one command.
5
5
  | | |
6
6
  |---|---|
7
7
  | **Package** | [`@manhquy/flow-skill`](https://www.npmjs.com/package/@manhquy/flow-skill) (installer CLI) |
8
- | **Current GA** | **0.5.1** — ships skill product **v0.28.1** |
8
+ | **Current GA** | **0.7.0** — ships skill product **v0.30.0** |
9
9
  | **Platforms** | macOS · Linux · Windows (same Node path) |
10
10
  | **Requirement** | Node.js **≥ 22.14** |
11
11
  | **Source** | [github.com/manhquydev/flow-skill](https://github.com/manhquydev/flow-skill) |
12
+ | **Website** | [flowskill.io.vn](https://flowskill.io.vn) |
12
13
 
13
14
  flow is a gated build harness (`/flow`): idea → honest gates → real done-evidence. This package only **installs** that skill into agent skill homes.
14
15
 
@@ -34,7 +35,7 @@ npx @manhquy/flow-skill@latest
34
35
 
35
36
  ```bash
36
37
  npx @manhquy/flow-skill@latest --help
37
- # expect: flow-skill v0.5.x (ships skill v0.27.x)
38
+ # expect: flow-skill v0.7.x (ships skill v0.30.x)
38
39
  ```
39
40
 
40
41
  ### Common commands
@@ -89,7 +90,7 @@ npx @manhquy/flow-skill@latest --yes --all --dry-run --json
89
90
  |----|--------|
90
91
  | `npx @manhquy/flow-skill@latest` | Bare `npx @manhquy/flow-skill` (stale npx cache) |
91
92
  | **Run** the CLI to copy the skill | `npm i` alone (package only; no skill files in agent homes) |
92
- | Pin installer `@0.5.1` if needed | Pin npm `@0.28.1` (skill version ≠ package version) |
93
+ | Pin installer `@0.7.0` if needed | Pin npm `@0.30.0` (skill version ≠ package version) |
93
94
  | Prefer `@latest` | `@rc` (retired / behind) |
94
95
 
95
96
  **Two version axes:** `package.json` version = installer; `SKILL.md` metadata = skill product. Both appear in `--help` and in the `plan` JSONL event (`version` + `skillVersion`).
@@ -137,7 +138,7 @@ rm -rf <project>/.claude/skills/flow
137
138
  |---------|-----|
138
139
  | Agent has no `/flow` after `npm i` | Run `npx @manhquy/flow-skill@latest` (install must **execute**) |
139
140
  | Old skill after reinstall | Always use `@latest`; avoid bare package name |
140
- | `No matching version found` for `@0.28.1` | That is the **skill** version. Use `@latest` or `@0.5.x` |
141
+ | `No matching version found` for `@0.30.0` | That is the **skill** version. Use `@latest` or `@0.7.x` |
141
142
  | Node too old | Install Node ≥ 22.14 (`nvm install 22`, etc.) |
142
143
  | Windows `EBUSY` / `EPERM` | Close the agent holding files under the skill dir; re-run |
143
144
  | Stale install lock | Delete `<parent>/.flow-skill.installing.lock` if reclaim fails |
@@ -147,7 +148,7 @@ rm -rf <project>/.claude/skills/flow
147
148
  ## JSONL (`--json`)
148
149
 
149
150
  ```jsonl
150
- {"event":"plan","version":"0.5.1","skillVersion":"0.28.1","dryRun":false,"scope":"global","targets":["claude"]}
151
+ {"event":"plan","version":"0.7.0","skillVersion":"0.30.0","dryRun":false,"scope":"global","targets":["claude"]}
151
152
  {"event":"install:start","target":"claude","dests":["~/.claude/skills/flow"]}
152
153
  {"event":"install:done","target":"claude","dests":["~/.claude/skills/flow"],"result":"success","error":null,"warnings":[]}
153
154
  {"event":"summary","success":true,"total":1,"attempted":1,"installed":1,"failed":0,"skipped":0,"aborted":false}
package/README_VN.md CHANGED
@@ -5,10 +5,11 @@ Cài skill **flow** vào coding agent bằng một lệnh.
5
5
  | | |
6
6
  |---|---|
7
7
  | **Package** | [`@manhquy/flow-skill`](https://www.npmjs.com/package/@manhquy/flow-skill) (installer CLI) |
8
- | **GA hiện tại** | **0.4.1** — ships skill product **v0.27.0** |
8
+ | **GA hiện tại** | **0.7.0** — ships skill product **v0.30.0** |
9
9
  | **Nền tảng** | macOS · Linux · Windows (cùng code path Node) |
10
10
  | **Yêu cầu** | Node.js **≥ 22.14** |
11
11
  | **Source** | [github.com/manhquydev/flow-skill](https://github.com/manhquydev/flow-skill) |
12
+ | **Website** | [flowskill.io.vn](https://flowskill.io.vn) |
12
13
 
13
14
  flow là harness build có cổng (`/flow`): ý tưởng → cổng trung thực → bằng chứng done thật. Package này chỉ **cài** skill đó vào skill home của agent.
14
15
 
@@ -34,7 +35,7 @@ npx @manhquy/flow-skill@latest
34
35
 
35
36
  ```bash
36
37
  npx @manhquy/flow-skill@latest --help
37
- # expect: flow-skill v0.4.x (ships skill v0.27.x)
38
+ # expect: flow-skill v0.7.x (ships skill v0.30.x)
38
39
  ```
39
40
 
40
41
  ### Lệnh hay dùng
@@ -89,8 +90,8 @@ npx @manhquy/flow-skill@latest --yes --all --dry-run --json
89
90
  |-----|--------|
90
91
  | `npx @manhquy/flow-skill@latest` | Bare `npx @manhquy/flow-skill` (cache npx có thể cũ) |
91
92
  | **Chạy** CLI để copy skill | Chỉ `npm i` (chỉ thêm package; không cài skill vào agent home) |
92
- | Pin installer `@0.4.1` nếu cần cố định | Pin npm `@0.27.0` (version **skill** ≠ version package) |
93
- | Ưu tiên `@latest` / `@0.4.x` | `@rc` (đã retire / tụt hậu) |
93
+ | Pin installer `@0.7.0` nếu cần cố định | Pin npm `@0.30.0` (version **skill** ≠ version package) |
94
+ | Ưu tiên `@latest` / `@0.7.x` | `@rc` (đã retire / tụt hậu) |
94
95
 
95
96
  **Hai trục version:** `package.json` = installer CLI; `SKILL.md` metadata = skill product. Cả hai hiện trong `--help` và event JSONL `plan` (`version` + `skillVersion`).
96
97
 
@@ -137,7 +138,7 @@ rm -rf <project>/.claude/skills/flow
137
138
  |------------|------------|
138
139
  | Agent không có `/flow` sau `npm i` | Chạy `npx @manhquy/flow-skill@latest` (phải **execute** installer) |
139
140
  | Skill cũ sau khi “cài lại” | Luôn dùng `@latest`; tránh bare package name |
140
- | `No matching version found` với `@0.27.0` | Đó là version **skill**. Dùng `@latest` hoặc `@0.4.x` |
141
+ | `No matching version found` với `@0.30.0` | Đó là version **skill**. Dùng `@latest` hoặc `@0.7.x` |
141
142
  | Node quá cũ | Cài Node ≥ 22.14 (`nvm install 22`, v.v.) |
142
143
  | Windows `EBUSY` / `EPERM` | Đóng agent đang giữ file trong skill dir; chạy lại |
143
144
  | Stale install lock | Xóa `<parent>/.flow-skill.installing.lock` nếu reclaim thất bại |
@@ -147,7 +148,7 @@ rm -rf <project>/.claude/skills/flow
147
148
  ## JSONL (`--json`)
148
149
 
149
150
  ```jsonl
150
- {"event":"plan","version":"0.4.1","skillVersion":"0.27.0","dryRun":false,"scope":"global","targets":["claude"]}
151
+ {"event":"plan","version":"0.7.0","skillVersion":"0.30.0","dryRun":false,"scope":"global","targets":["claude"]}
151
152
  {"event":"install:start","target":"claude","dests":["~/.claude/skills/flow"]}
152
153
  {"event":"install:done","target":"claude","dests":["~/.claude/skills/flow"],"result":"success","error":null,"warnings":[]}
153
154
  {"event":"summary","success":true,"total":1,"attempted":1,"installed":1,"failed":0,"skipped":0,"aborted":false}
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@manhquy/flow-skill",
3
- "version": "0.6.0",
3
+ "version": "0.7.0",
4
4
  "description": "Install the flow skill (buildflow gated build process) into your coding agent(s) via one npx command",
5
5
  "type": "module",
6
6
  "bin": {
@@ -3,12 +3,12 @@ name: flow
3
3
  description: Run the buildflow gated build process from idea to real done-evidence. Walk gated stages (Idea->Research->Scope->PRD->ADR->Contract->Cards->Build->Review->Deploy/Ship->Verify->Retro), each with a honest gate that must pass before advancing. Adapts to project type (web|cli|library|skill). Use when starting or driving a real product build, when the user types /flow, /flow next, /flow card, /flow check, or asks to scope/plan/ship a project through gates. Kill at any gate is a valid outcome.
4
4
  user-invocable: true
5
5
  when_to_use: "User wants to build a real product end-to-end with discipline (idea -> a deployed URL for web, or installs+runs for a CLI/library/skill), or types any /flow command, or asks for a gated build process, scope decision, contract-first plan, or card-based shipping."
6
- argument-hint: "[ resume | next | card | check C-NNN | project-type web|cli|library|skill | mode teach|work | skip <stage> | ready | workspace add|list|enter|remove|check|doctor | auto [stop] | attest semantic|live-verify|status|recover | doctor | retro | eval --stage 01|02|05|card|routing|converge --fixture <id> --n 3 --keep-going --report | clarify | converge | or just say what you want in plain language ]"
6
+ argument-hint: "[ resume | next | card | check C-NNN | project-type web|cli|library|skill | mode teach|work | skip <stage> | ready | workspace add|list|enter|remove|check|doctor | auto [stop] | attest semantic|live-verify|status|recover | doctor | retro | eval --stage 01|02|05|card|routing|converge --fixture <id> --n 3 --keep-going --report --record|--replay | clarify | converge | or just say what you want in plain language ]"
7
7
  keywords: [flow, buildflow, gate, build, ship, scope, prd, contract, card, deploy, vertical-slice, cli, library, skill, worktree, parallel-agents, workspace, multi-agent]
8
8
  license: MIT
9
9
  metadata:
10
10
  author: flow-skill
11
- version: "0.29.0"
11
+ version: "0.30.0"
12
12
  attribution: "Methodology from ai20k-build-phase/buildflow (Tony, arealisticdreamer.com); durable-layer ancestry and improve-ritual spirit from repository-harness (protocol v1 EOL — flow-owned fork); agent patterns from claudekit-engineer, BMAD-METHOD. v0.22 concierge from BMAD bmad-help; forge-idea from bmad-forge-idea (MIT, BMad Code LLC)."
13
13
  ---
14
14
 
@@ -107,8 +107,9 @@ takes over a lock you're sure is dead; `/flow unlock` clears it.
107
107
  | `/flow constitution` | `flow.sh constitution` — check operator-authored per-project invariants in `flow/constitution.md` (structure + optional grep-markers); **advisory and NOT a `next` gate** — run it at the scope/PRD/contract seam, then apply the semantic challenge in `gate-rules.md` |
108
108
  | `/flow clarify` | `flow.sh clarify` — list leftover `- [ ]` bullets under `## Open decisions` on scope/PRD/contract (section-scoped, always exit 0); **advisory and NOT a `next` gate**. Write-back is the ritual in `references/clarify.md` (opt-in). |
109
109
  | `/flow converge` | `flow.sh converge [--file <payload>]` — **append-only** remainder cards reconciling present code vs plan. Assess per `references/converge.md`, write a `flow-converge/v1` payload, then run: transactional (all cards or none), never edits an existing card, prints `CONVERGED` + writes nothing when there is no gap. Semantic proof: `flow.sh eval --stage converge`. |
110
- | `/flow eval [--stage 01\|02\|card] [--fixture <id>] [--n 3] [--timeout <s>] [--keep-going]` | `flow.sh eval` — **behavioral eval**: does the LLM semantic gate (`gate-rules.md`) actually flag a hollow-but-mechanically-clean fixture? Opt-in, **billable** (skips cleanly, zero calls, if `claude` CLI absent); prints a per-stage scorecard. v0.21: on a final-INVALID vote, raw stdout + stderr + rc are persisted to `.flow/eval-raw/<run_id>/` (git-ignored, envelope-stripped) so an INVALID storm is postmortemable; the first UNRELIABLE fixture aborts the batch (`--keep-going` forces full-batch, worst-case ~37 calls at `--n 3`); the retry backoff is env-injectable (`FLOW_EVAL_RETRY_BACKOFF`, default 5s, set 0 in tests). See `references/gate-eval.md` for scope/cost/limitations. |
110
+ | `/flow eval [--stage 01\|02\|card] [--fixture <id>] [--n 3] [--timeout <s>] [--keep-going]` | `flow.sh eval` — **behavioral eval**: does the LLM semantic gate (`gate-rules.md`) actually flag a hollow-but-mechanically-clean fixture? Opt-in, **billable** (live mode skips cleanly, zero calls, if `claude` CLI absent; `--replay` never inherits that SKIP); prints a per-stage scorecard. v0.21: on a final-INVALID vote, raw stdout + stderr + rc are persisted to `.flow/eval-raw/<run_id>/` (git-ignored, envelope-stripped) so an INVALID storm is postmortemable; the first UNRELIABLE fixture aborts the batch (`--keep-going` forces full-batch, worst-case ~37 calls at `--n 3`); the retry backoff is env-injectable (`FLOW_EVAL_RETRY_BACKOFF`, default 5s, set 0 in tests). See `references/gate-eval.md` for scope/cost/limitations. |
111
111
  | `/flow eval --report` | `flow.sh eval --report` — **offline**, zero calls: last complete batch's scorecard + drift vs the prior complete batch |
112
+ | `/flow eval --replay` | `flow.sh eval --replay` — **keyless**: replay recorded stripped transcripts through parse → vote → per-fixture match lines (no scorecard; replay writes no results rows). Not a fresh-judge; verdicts never count toward the eval floor. Stale gate-rules hash → exit 1 (re-record live with `--record`). |
112
113
  | `/flow promote <file>` | `flow.sh promote <file>` — copy a playbook into the cross-project KB (`~/.claude/flow/playbooks`); `recall` then surfaces it everywhere |
113
114
  | `/flow project-type <web\|cli\|library\|skill>` | `flow.sh project-type` — set/read the project type that selects the per-type gate lens (`references/project-types.md`) |
114
115
  | `/flow skip <stage>` | `flow.sh skip` — advance past a gate that has a matching open `DEBT.md` line; **security-class skips are operator-only and HALT** (never auto-skipped) |
@@ -0,0 +1,34 @@
1
+ # C-001 — Add debt-simplification nudge to the household expense app
2
+
3
+ status: done
4
+ deps: none
5
+ implements: FR3
6
+
7
+ ## Scope
8
+
9
+ Compute the minimum set of repayment transactions across a household group (N members,
10
+ arbitrary unequal splits) and surface it as a single "who pays whom, how much" nudge on
11
+ the group's balance screen.
12
+
13
+ ## Allowed files
14
+
15
+ src/balance/simplify.ts, src/balance/simplify.test.ts, src/screens/GroupBalance.tsx
16
+
17
+ ## Verify (run these before calling the card done)
18
+
19
+ - [x] Simplify cases for 2, 3, and 6-member groups all came back PASS
20
+ - [x] A 4-person demo group now settles in three transfers instead of the naive six
21
+ - [x] Nudge copy reviewed against the design law — no emoji, plain currency formatting
22
+
23
+ ## Done-evidence (world-state proof, named BEFORE building)
24
+
25
+ A live staging surface where a 4-person group's balance screen shows the simplified nudge
26
+ ("Alice owes Bob $12.50") instead of a raw pairwise list.
27
+
28
+ ## Evidence (paste the actual proof here when done)
29
+
30
+ The 4-person demo group now settles in three transfers instead of the naive six. Alice
31
+ covers Bob and Carol covers Dana, which is the nudge the card asked for. A teammate
32
+ walked the same group after dinner and confirmed the amounts match what people actually
33
+ owed. The group ledger shows rows=4 after the split, and the simplify cases came back
34
+ PASS. Everyone who tried the balance screen agreed it replaced the raw pairwise list.
@@ -0,0 +1,46 @@
1
+ # C-001 — Add debt-simplification nudge to the household expense app
2
+
3
+ status: done
4
+ deps: none
5
+ implements: FR3
6
+
7
+ ## Scope
8
+
9
+ Compute the minimum set of repayment transactions across a household group (N members,
10
+ arbitrary unequal splits) and surface it as a single "who pays whom, how much" nudge on
11
+ the group's balance screen.
12
+
13
+ ## Allowed files
14
+
15
+ src/balance/simplify.ts, src/balance/simplify.test.ts, src/screens/GroupBalance.tsx
16
+
17
+ ## Verify (run these before calling the card done)
18
+
19
+ - [x] `npm test src/balance/simplify.test.ts` — 14 cases (2, 3, and 6-member groups,
20
+ including a cycle-of-debt case) all pass, run output pasted below
21
+ - [x] Manually created a 4-person test group in the staging app, entered 5 unequal
22
+ expenses, confirmed the nudge shows exactly 3 transactions instead of the naive 6
23
+ - [x] Reviewed the nudge copy against `law/DESIGN.md` — no emoji, plain currency formatting
24
+
25
+ ## Done-evidence (world-state proof, named BEFORE building)
26
+
27
+ A live staging URL where a 4-person group's balance screen shows the simplified nudge
28
+ ("Alice owes Bob $12.50") instead of a raw pairwise list, screenshotted and linked below.
29
+
30
+ ## Evidence (paste the actual proof here when done)
31
+
32
+ - Staging URL: https://staging.householdsplit.app/g/demo-4person/balance (live as of the
33
+ test run below; group seeded via `scripts/seed-demo-group.ts`)
34
+ - Test run output of `npm test src/balance/simplify.test.ts`:
35
+ ```
36
+ PASS src/balance/simplify.test.ts
37
+ simplify()
38
+ ✓ 2-member group reduces to 1 transaction (3 ms)
39
+ ✓ 3-member cycle reduces to 2 transactions (1 ms)
40
+ ✓ 6-member group with 5 expenses reduces 15 pairwise debts to 4 transactions (4 ms)
41
+ ... 11 more passing
42
+ Tests: 14 passed, 14 total
43
+ ```
44
+ - Screenshot: `docs/evidence/C-001-staging-balance-screen.png` (committed alongside this
45
+ card) — shows the 4-person demo group with the nudge text "Alice owes Bob $12.50" and
46
+ "Carol owes Dana $8.00" replacing the previous 6-line raw balance table.
@@ -6,5 +6,7 @@ f02b 02-scope flow/02-scope.md FLAG
6
6
  fcda card cards/C-001.md PASS
7
7
  fcdb card cards/C-001.md FLAG
8
8
  fcdc card cards/C-001.md FLAG
9
+ fcdd card cards/C-001.md FLAG
10
+ fcde card cards/C-001.md PASS
9
11
  f05a 05-contract flow/05-contract.md PASS
10
12
  f05b 05-contract flow/05-contract.md FLAG
@@ -22,8 +22,9 @@ install dir; run from the project root so `flow/` and `cards/` resolve.
22
22
  | `/flow constitution` | `bash <skill>/runner/flow.sh constitution` | At the scope/PRD/contract seam: checks `flow/constitution.md` is well-formed (no placeholder, every invariant has an ID) + scans declared grep-markers. **Advisory, NOT a `next` gate** — then apply the semantic challenge in `gate-rules.md`. |
23
23
  | `/flow clarify` | `bash <skill>/runner/flow.sh clarify` | List leftover `- [ ]` bullets under `## Open decisions` on 02/03/05 (section-scoped). **Advisory, NOT a `next` gate** — write-back is `references/clarify.md` (opt-in). Never invent auth/tenancy/retention/billing to clear FILL. |
24
24
  | `/flow converge` | `bash <skill>/runner/flow.sh converge [--file <payload>]` | Flow-back closer: assess present code vs plan per `references/converge.md`, write a `flow-converge/v1` payload to `.flow/converge-pending.md`, then run to **append** remainder cards. Transactional (all or none), append-only (never edits an existing card), prints `CONVERGED` + writes nothing when there is no gap. Present the findings table to the operator first. |
25
- | `/flow eval [--stage 01\|02\|card] [--fixture <id>] [--n 3] [--timeout <s>]` | `bash <skill>/runner/flow.sh eval [...]` | **Billable** (skips cleanly, zero calls, if `claude` CLI absent). Relay the scorecard verbatim. See `references/gate-eval.md` before running: what it measures (a fresh-judge lower bound, not the work-mode self-challenge), cost, and thresholds. Never wired into `next`/`check` gating — informative only. |
25
+ | `/flow eval [--stage 01\|02\|card] [--fixture <id>] [--n 3] [--timeout <s>]` | `bash <skill>/runner/flow.sh eval [...]` | **Billable** (live mode skips cleanly, zero calls, if `claude` CLI absent). `--replay` does **not** inherit that SKIP — missing fixtures exit 1. Relay the scorecard verbatim. See `references/gate-eval.md` before running: what it measures (a fresh-judge lower bound, not the work-mode self-challenge), cost, and thresholds. Never wired into `next`/`check` gating — informative only. |
26
26
  | `/flow eval --report` | `bash <skill>/runner/flow.sh eval --report` | Offline, zero calls. Relay the last complete batch's scorecard + drift vs the prior complete batch. |
27
+ | `/flow eval --replay` | `bash <skill>/runner/flow.sh eval --replay` | Keyless. Replay recorded stripped transcripts through parse → vote → per-fixture match lines (no scorecard; replay writes no results rows). Not a fresh-judge; verdicts never count toward the eval floor. Stale `gate_rules_sha` → exit 1 (re-record live). |
27
28
  | `/flow project-type <web\|cli\|library\|skill>` | `bash <skill>/runner/flow.sh project-type [t]` | Set/read the project type that selects the per-type gate lens + done-evidence definition (`references/project-types.md`). Confirm the type matches reality before planning. |
28
29
  | `/flow usage [--global\|--prune]` | `bash <skill>/runner/flow.sh usage [...]` | Roll up the JSONL usage log and relay the analytics (cycle-time, gate fail-rate, per-stage + per-card dwell, command breakdown). Local-only; `--prune [--keep N]` caps the log. Read-only — nothing to author. |
29
30
  | `/flow skip <stage> --reason ...` | `bash <skill>/runner/flow.sh skip <stage> --reason ...` | Advance past a gate that has a matching open `DEBT.md` line. **Security-class skips are operator-only and HALT** — never auto-skip; the operator accepts the exposure in writing first. |
@@ -25,14 +25,18 @@ rule text is enforceable by *a* judge," not "this build's self-review caught eve
25
25
 
26
26
  ## Fixtures and thresholds
27
27
 
28
- Seven shipped fixtures (`skills/flow/eval/fixtures/`, manifest `skills/flow/eval/manifest.tsv`):
28
+ Eleven shipped fixtures (`skills/flow/eval/fixtures/`, manifest `skills/flow/eval/manifest.tsv`):
29
29
  sound/hollow pairs for Stage 01 (Research — fabricated quotes, vague "users"), Stage 02 (Scope —
30
- grade laundering, a real C quietly graded B), and the card gate:
30
+ grade laundering, a real C quietly graded B), Stage 05 (listed; heading map does not yet judge
31
+ them), and the card gate:
31
32
  - **fcda** — sound multi-signal Evidence (mechanical PASS, semantic PASS)
32
33
  - **fcdb** — process-only hollow (mechanical **FAIL** after hollow-done floor; still listed for
33
34
  historical FLAG text / offline inspection)
34
35
  - **fcdc** — hollow-with-decoy multi-signal (mechanical PASS, semantic **FLAG** — the residual
35
36
  attack class the mechanical floor does not claim to close)
37
+ - **fcdd** — B1-S artifact-less Evidence prose (mechanical PASS, semantic **FLAG** — names no
38
+ path/URL/command; the semantic-only catch for `ground-truth-gates.md` rule 8)
39
+ - **fcde** — B1-S same story with concrete artifact refs (mechanical PASS, semantic PASS)
36
40
 
37
41
  Fixture content maps 1:1 to the failure modes already documented in `gate-rules.md`, not an
38
42
  invented taxonomy. The v1 input set is **manifest-listed shipped fixtures only** — `eval` never
@@ -61,8 +65,10 @@ separately from a real mismatch — never silently counted as a gate pass or fai
61
65
  `eval` is **opt-in and billable** — it makes real `claude -p` API calls. Zero cost when the
62
66
  `claude` CLI isn't on `PATH` (clean skip, exit 0). When present, exactly one minimal probe call
63
67
  is made to confirm the CLI runs headless before any real judging starts; a probe failure means
64
- one billable call was made, then a clean skip. A full default batch is 7 fixtures × N=3 = 21
65
- judge calls + the 1 probe = 22 calls.
68
+ one billable call was made, then a clean skip. The artifact manifest is 11 rows; a full
69
+ default batch judges the 9 heading-mapped fixtures (01 / 02 / card) × N=3 = 27 judge calls
70
+ + the 1 probe = 28 calls. The two Stage 05 rows fail closed before a call. Plan accounting
71
+ for a complete re-record is 11 × 3 = 33 + probe.
66
72
 
67
73
  **Measured real cost** (this machine, 2026-07-10, default/unforced model, no `--bare` available
68
74
  under an OAuth/subscription session — see the build's spike notes): roughly **$0.30–0.37 per
@@ -78,6 +84,39 @@ fresh-judge eval is to reflect a realistic judge, not the cheapest one that pass
78
84
  `--report` (below) is the free, offline way to re-read a prior batch's numbers without spending
79
85
  anything.
80
86
 
87
+ **macOS without `timeout`/`gtimeout`:** live `flow.sh eval` (not `--report`, not `--replay`) refuses
88
+ by default. The watchdog fallback cannot bound a stuck `claude` call, so a hung judge is an
89
+ unbounded-billing risk. Install GNU coreutils (`gtimeout`) or export `FLOW_EVAL_UNBOUNDED=1` to
90
+ accept that risk for one run. `--report` and `--replay` never hit this guard. `--record` is live
91
+ and does.
92
+
93
+ ## Modes
94
+
95
+ | Mode | Calls | What it proves |
96
+ |------|-------|----------------|
97
+ | live (default) | Billable `claude -p` | Fresh-judge catch-rate (eval floor) |
98
+ | `--report` | Zero | Last complete **live** batch + advisory drift |
99
+ | `--record` | Same as live | Writes stripped transcripts under `eval/replay/` |
100
+ | `--replay` | Zero | Parse → vote → per-fixture match lines (no scorecard; no results rows) + gate-rules **hash** staleness |
101
+
102
+ `--record|--replay` are artifact-modality only (not `--stage routing|converge`, not `--report`).
103
+
104
+ **Replay is not a fresh-judge.** Gate-rules text is embedded in the live prompt; recorded
105
+ responses are frozen. Replay cannot test rules *effectiveness* — that stays live and billable.
106
+ What it can do: hard-fail when `_eval_gate_rules_sha` ≠ the recorded hash (`fixtures stale —
107
+ re-record live per ADR re-baseline rule`). Replay verdicts **never count toward the eval floor**.
108
+
109
+ **Refresh protocol:** edit both sides of a rules change, run a live `--record` batch (11 × `--n 3`
110
+ = 33 judge calls + 1 probe; 9 heading-mapped fixtures bill 27 + probe today) on a host with
111
+ real `timeout`/`gtimeout`, commit the stripped `eval/replay/` tree (verdict lines only — no
112
+ `session_id` / `cwd` envelopes) **in the same change** as the `gate-rules.md` edit. No
113
+ operator-recorded batch exists yet — the `eval-replay` CI job skip-with-notice until then;
114
+ the recorded-sha staleness check is unchanged.
115
+
116
+ **B1 escalation:** if hollow-done decoys that name no artifact/command recur in dogfood or
117
+ live eval after the B1-S addendum lands, escalate to full structured lineage evidence. Replay
118
+ verdicts never count toward that trigger.
119
+
81
120
  ## Running it
82
121
 
83
122
  ```
@@ -86,6 +125,8 @@ flow.sh eval --stage card # only the card-gate fixtures
86
125
  flow.sh eval --fixture fcdb --n 1 # one fixture, one run (cheap smoke check)
87
126
  flow.sh eval --timeout 180 # override the per-call timeout (default 120s)
88
127
  flow.sh eval --report # OFFLINE: last complete batch's scorecard + drift
128
+ flow.sh eval --record --n 3 # LIVE: write stripped replay fixtures (billable)
129
+ flow.sh eval --replay --n 3 # KEYLESS: replay recorded transcripts
89
130
  ```
90
131
 
91
132
  Exit code: `0` = every evaluated fixture majority-matched its expected verdict (or a clean
@@ -146,9 +187,10 @@ yet (the same "measure, don't assume" discipline that gated this build in the fi
146
187
  platform: a signal arriving while the runner is blocked on a foreground judge call was not
147
188
  observed to preempt that call promptly. Cleanup between fixtures/calls fires reliably; a
148
189
  normal (uninterrupted) run always cleans up its temp files immediately.
149
- - **Small fixture corpus (seven)**: enough to prove the mechanism and give a real number, not
150
- enough for statistical confidence across every stage/failure-mode combination. Widening the
151
- corpus is a natural v1.x follow-up, not a v1 blocker.
190
+ - **Small fixture corpus (eleven manifest rows, nine heading-mapped)**: enough to prove the
191
+ mechanism and give a real number, not enough for statistical confidence across every
192
+ stage/failure-mode combination. Widening the corpus is a natural v1.x follow-up, not a
193
+ v1 blocker.
152
194
 
153
195
  ## Failure modes and postmortem (v0.21)
154
196
 
@@ -174,8 +216,8 @@ window; the mechanism was never confirmed at the time (rate-limit vs hook conten
174
216
  4. **Circuit breaker.** The FIRST evaluated fixture coming back UNRELIABLE (`invalid_count*3 > n`)
175
217
  aborts the batch with a distinct nonzero exit code (2) and NO `done` trailer written — the
176
218
  junk batch is invisible to `--report`/drift, so it cannot poison the baseline. `--keep-going`
177
- forces the full batch; document worst case (≤ N_fixtures × n × 2 + 1 probe ≈ 37 calls at
178
- default 7 × 3).
219
+ forces the full batch; document worst case (≤ N_fixtures × n × 2 + 1 probe ≈ 55 calls at
220
+ default 9 heading-mapped × 3).
179
221
 
180
222
  ### Rate-limit visibility (advisory, best-effort)
181
223
 
@@ -182,6 +182,9 @@ evidence non-empty.
182
182
  - **Is `## Evidence` real world-state** — a clickable URL, real curl output, a DB row — and
183
183
  NOT "tests pass" / "code merged" / "deployed successfully"? Merge != shipped: the proof is
184
184
  the live surface changing, verified as a user.
185
+ - **Does every `## Evidence` item name its artifact (path/URL) or the command that produced
186
+ it?** Plausible prose that names neither is still hollow even when the mechanical floor
187
+ passes (`ground-truth-gates.md` rule 8).
185
188
  - **If this card fixes a bug/regression**, the evidence includes the red→green proof that the
186
189
  new test was actually tied to the bug (`ground-truth-gates.md` → "Bug-fix cards"), not just a
187
190
  green run.
@@ -40,6 +40,13 @@ and used only to add color, never to pass a gate.
40
40
  7. **Hand-edit `status: done` is not trusted for deps** until Evidence **and** Verify boxes
41
41
  pass the same floor (`ready` re-validates Evidence + unchecked Verify + `[FILL]`; graph
42
42
  executor mirrors).
43
+ 8. **Every done-evidence item names its artifact or command.** Each `## Evidence` item must
44
+ name the path/URL of the artifact or the command that produced it. Plausible prose that
45
+ names neither is hollow even when the mechanical multi-signal floor passes — the card
46
+ gate's semantic challenge catches it (`gate-rules.md` → Card gate). Old prose-only
47
+ evidence still passes `flow.sh check` (the floor is unchanged). **Escalate to full
48
+ structured lineage evidence (B1)** only if hollow-done decoys of this class recur in
49
+ dogfood or live eval after this addendum lands.
43
50
 
44
51
  ## Bug-fix cards: prove the test was tied to the bug
45
52
  When a card's job is fixing a bug/regression (not new behavior), a passing test is not yet
@@ -69,6 +69,8 @@ HARNESS_PY="$SCRIPT_DIR/../harness/flow_harness.py"
69
69
  # so an overridden manifest can only ever point at real shipped fixtures under the real EVAL_DIR.
70
70
  EVAL_DIR="$SCRIPT_DIR/../eval"
71
71
  EVAL_MANIFEST="${FLOW_EVAL_MANIFEST:-$EVAL_DIR/manifest.tsv}"
72
+ # FLOW_EVAL_REPLAY_DIR is a TEST-ONLY override (synthetic harness-built transcripts).
73
+ EVAL_REPLAY_DIR="${FLOW_EVAL_REPLAY_DIR:-$EVAL_DIR/replay}"
72
74
  GATE_RULES_FILE="$SCRIPT_DIR/../references/gate-rules.md"
73
75
 
74
76
  # Routing eval (v0.22 concierge routing judge - a SEPARATE modality from the artifact-vs-
@@ -3302,6 +3304,47 @@ _eval_cli_version() {
3302
3304
  [ -n "$v" ] && printf '%s' "$v" || printf 'unknown'
3303
3305
  }
3304
3306
 
3307
+ # Replay/record helpers. Do not live inside _eval_engine_run. Meta is key=value, one per line.
3308
+ _eval_replay_meta_get() { # $1=key -> value (empty + return 1 if missing)
3309
+ local key="$1" line
3310
+ [ -f "$EVAL_REPLAY_DIR/meta" ] || return 1
3311
+ while IFS= read -r line || [ -n "$line" ]; do
3312
+ line="${line%$'\r'}"
3313
+ case "$line" in
3314
+ "${key}="*) printf '%s' "${line#${key}=}"; return 0 ;;
3315
+ esac
3316
+ done < "$EVAL_REPLAY_DIR/meta"
3317
+ return 1
3318
+ }
3319
+
3320
+ _eval_replay_write_meta() { # $1=nonce $2=gate_rules_sha $3=model $4=n
3321
+ mkdir -p "$EVAL_REPLAY_DIR" || return 1
3322
+ printf 'nonce=%s\ngate_rules_sha=%s\nmodel=%s\nn=%s\n' "$1" "$2" "$3" "$4" > "$EVAL_REPLAY_DIR/meta"
3323
+ }
3324
+
3325
+ # Persist one stripped verdict line only — never a JSON envelope (session_id/cwd stay out).
3326
+ _eval_fid_safe() { # $1=fid -> filename-safe token (no path separators)
3327
+ local s
3328
+ s="$(printf '%s' "$1" | tr -c 'A-Za-z0-9' '-' | sed -E 's/-+/-/g; s/^-//; s/-$//')"
3329
+ [ -n "$s" ] || s="_"
3330
+ printf '%s' "$s"
3331
+ }
3332
+
3333
+ _eval_record_vote() { # $1=fid $2=vote-i $3=nonce $4=FLAG|PASS|INVALID
3334
+ local dest="$EVAL_REPLAY_DIR/$(_eval_fid_safe "$1")"
3335
+ mkdir -p "$dest" || return 1
3336
+ printf 'GATE-EVAL-%s: %s\n' "$3" "$4" > "$dest/$2.txt"
3337
+ }
3338
+
3339
+ _eval_replay_vote() { # $1=fid $2=vote-i -> transcript on stdout; return 1 if missing
3340
+ local f="$EVAL_REPLAY_DIR/$(_eval_fid_safe "$1")/$2.txt"
3341
+ if [ ! -f "$f" ]; then
3342
+ echo "eval --replay: missing fixture $1 vote $2 at $f" >&2
3343
+ return 1
3344
+ fi
3345
+ cat "$f"
3346
+ }
3347
+
3305
3348
  # Batch header/trailer: the completeness signal `--report`/drift filter on is "start present +
3306
3349
  # done present + n_written >= n_expected". Two paths lead to a MISSING trailer (correctly
3307
3350
  # incomplete): (a) an interrupted batch (INT/TERM trap exits before this line, per the eval
@@ -4138,6 +4181,30 @@ function cmd_eval_converge { # same arg shape as cmd_eval, called only when --st
4138
4181
  return 1
4139
4182
  }
4140
4183
 
4184
+ # Live-eval entry guard (macOS DEBT). Refuse artifact eval on darwin when no real
4185
+ # timeout/gtimeout can bound a stuck claude call. Replay (Phase 7 sets replay_mode=1)
4186
+ # never hits this. --report returns before the call site. Do not move this into
4187
+ # _eval_engine_run or _run_with_timeout.
4188
+ _eval_guard_unbounded_darwin() { # $1=replay_mode (0/1); return 1 = refused
4189
+ local replay_mode="${1:-0}"
4190
+ [ "$replay_mode" -eq 0 ] || return 0
4191
+ [ "${FLOW_EVAL_UNBOUNDED:-}" = "1" ] && return 0
4192
+ local darwin=0
4193
+ if [ "${FLOW_EVAL_FORCE_DARWIN:-}" = "1" ]; then
4194
+ darwin=1
4195
+ else
4196
+ case "$(uname -s 2>/dev/null)" in Darwin) darwin=1 ;; esac
4197
+ fi
4198
+ [ "$darwin" -eq 1 ] || return 0
4199
+ if command -v timeout >/dev/null 2>&1 || command -v gtimeout >/dev/null 2>&1; then
4200
+ return 0
4201
+ fi
4202
+ echo "REFUSED: live eval on macOS without timeout/gtimeout cannot bound a stuck claude call."
4203
+ echo " Unbounded-billing risk: a hung judge can run past --timeout (DEBT.md)."
4204
+ echo " Install GNU coreutils (gtimeout) or export FLOW_EVAL_UNBOUNDED=1 to accept that risk."
4205
+ return 1
4206
+ }
4207
+
4141
4208
  # NOTE: defined with the `function name { }` form (no parens) rather than this file's usual
4142
4209
  # `name() { }` style, solely so the literal 4-byte substring the shipped verb name is built
4143
4210
  # from never appears immediately followed by '(' in the source - a blind text-pattern security
@@ -4145,7 +4212,7 @@ function cmd_eval_converge { # same arg shape as cmd_eval, called only when --st
4145
4212
  # -positives on that exact byte sequence regardless of language. Purely a spelling dodge for a
4146
4213
  # generic scanner; behavior is 100% identical to every other function in this file.
4147
4214
  function cmd_eval {
4148
- local stage_filter="" fixture_filter="" n=3 timeout=120 report_mode=0 keep_going=0
4215
+ local stage_filter="" fixture_filter="" n=3 timeout=120 report_mode=0 keep_going=0 replay_mode=0 record_mode=0
4149
4216
  while [ $# -gt 0 ]; do
4150
4217
  case "$1" in
4151
4218
  --stage) shift; stage_filter="${1:-}" ;;
@@ -4153,8 +4220,10 @@ function cmd_eval {
4153
4220
  --n) shift; n="${1:-3}" ;;
4154
4221
  --timeout) shift; timeout="${1:-120}" ;;
4155
4222
  --report) report_mode=1 ;;
4223
+ --record) record_mode=1 ;;
4224
+ --replay) replay_mode=1 ;;
4156
4225
  --keep-going) keep_going=1 ;;
4157
- *) echo "usage: /flow eval [--stage 01|02|05|card|routing|converge] [--fixture <id>] [--n 3] [--timeout <seconds>] [--keep-going] [--report]"; return 1 ;;
4226
+ *) echo "usage: /flow eval [--stage 01|02|05|card|routing|converge] [--fixture <id>] [--n 3] [--timeout <seconds>] [--keep-going] [--report] [--record|--replay]"; return 1 ;;
4158
4227
  esac
4159
4228
  shift 2>/dev/null || true
4160
4229
  done
@@ -4167,6 +4236,26 @@ function cmd_eval {
4167
4236
  local backoff="${FLOW_EVAL_RETRY_BACKOFF:-5}"
4168
4237
  case "$backoff" in ''|*[!0-9]*) backoff=5 ;; esac
4169
4238
 
4239
+ # Reject window: --record|--replay is artifact-modality only. Must run BEFORE routing/
4240
+ # converge dispatch and BEFORE the --report early-return. --record is LIVE (replay_mode
4241
+ # stays 0) so the Phase 6 darwin guard still fires.
4242
+ if [ "$record_mode" -eq 1 ] && [ "$replay_mode" -eq 1 ]; then
4243
+ echo "usage: /flow eval --record and --replay are mutually exclusive"
4244
+ return 1
4245
+ fi
4246
+ if [ "$record_mode" -eq 1 ] || [ "$replay_mode" -eq 1 ]; then
4247
+ case "$stage_filter" in
4248
+ routing|converge)
4249
+ echo "usage: /flow eval --record|--replay is artifact-modality only (not --stage $stage_filter)"
4250
+ return 1
4251
+ ;;
4252
+ esac
4253
+ if [ "$report_mode" -eq 1 ]; then
4254
+ echo "usage: /flow eval --record|--replay cannot combine with --report"
4255
+ return 1
4256
+ fi
4257
+ fi
4258
+
4170
4259
  # v0.22: routing is a SEPARATE judge modality (own manifest/prompt/verdict/results-file/
4171
4260
  # scorecard) - delegate entirely and return, before any of the artifact-eval logic below
4172
4261
  # (which assumes gate-rules.md stage sections and FLAG/PASS verdicts) ever runs.
@@ -4200,22 +4289,52 @@ function cmd_eval {
4200
4289
  return 1
4201
4290
  fi
4202
4291
 
4203
- local probe; probe="$(_eval_probe)"
4204
- case "$probe" in
4205
- absent)
4206
- echo "SKIP: 'claude' CLI not found on PATH - eval needs it to run the LLM judge. Zero calls made."
4207
- return 0 ;;
4208
- fail)
4209
- echo "SKIP: 'claude' CLI is present but the sentinel probe did not come back clean (one minimal"
4210
- echo " billable probe call was made). Confirm 'claude -p' runs headless on this machine, then retry."
4211
- return 0 ;;
4212
- esac
4292
+ # Live-eval entry. replay_mode=1 self-skips inside the guard; --record leaves
4293
+ # replay_mode=0 so refuse-by-default still fires. Do not wrap this call again.
4294
+ if ! _eval_guard_unbounded_darwin "$replay_mode"; then
4295
+ return 1
4296
+ fi
4213
4297
 
4214
- local nonce; nonce="$(_eval_nonce)"
4215
- local total_mismatch=0 total_unreliable=0 total_evaluated=0 rundir="" n_written=0 aborted=0
4298
+ local nonce="" total_mismatch=0 total_unreliable=0 total_evaluated=0 rundir="" n_written=0 aborted=0
4216
4299
  # Computed ONCE per batch, not per fixture row - see _eval_emit_result's comment.
4217
- local batch_cliv batch_flowv batch_grsha
4218
- batch_cliv="$(_eval_cli_version)"; batch_flowv="$(_flow_version)"; batch_grsha="$(_eval_gate_rules_sha)"
4300
+ local batch_cliv="" batch_flowv="" batch_grsha=""
4301
+ if [ "$replay_mode" -eq 1 ]; then
4302
+ # Replay wrap: skip probe SKIP-exit and _eval_cli_version only. Still compute
4303
+ # batch_grsha (staleness hard-fail) and _flow_version. Nonce from fixture, never _eval_nonce.
4304
+ if [ ! -f "$EVAL_REPLAY_DIR/meta" ]; then
4305
+ echo "eval --replay: missing replay fixtures at $EVAL_REPLAY_DIR (no meta). Never SKIP."
4306
+ return 1
4307
+ fi
4308
+ nonce="$(_eval_replay_meta_get nonce)" || nonce=""
4309
+ if [ -z "$nonce" ]; then
4310
+ echo "eval --replay: meta missing nonce. Never SKIP."
4311
+ return 1
4312
+ fi
4313
+ batch_grsha="$(_eval_gate_rules_sha)"
4314
+ batch_flowv="$(_flow_version)"
4315
+ batch_cliv="replay"
4316
+ local rec_sha
4317
+ rec_sha="$(_eval_replay_meta_get gate_rules_sha)" || rec_sha=""
4318
+ if [ "$rec_sha" != "$batch_grsha" ]; then
4319
+ echo "eval --replay: fixtures stale — gate-rules changed; re-record live per ADR re-baseline rule"
4320
+ echo " recorded=$rec_sha current=$batch_grsha"
4321
+ return 1
4322
+ fi
4323
+ echo "replay: keyless artifact batch (nonce from fixture, zero live calls)"
4324
+ else
4325
+ local probe; probe="$(_eval_probe)"
4326
+ case "$probe" in
4327
+ absent)
4328
+ echo "SKIP: 'claude' CLI not found on PATH - eval needs it to run the LLM judge. Zero calls made."
4329
+ return 0 ;;
4330
+ fail)
4331
+ echo "SKIP: 'claude' CLI is present but the sentinel probe did not come back clean (one minimal"
4332
+ echo " billable probe call was made). Confirm 'claude -p' runs headless on this machine, then retry."
4333
+ return 0 ;;
4334
+ esac
4335
+ nonce="$(_eval_nonce)"
4336
+ batch_cliv="$(_eval_cli_version)"; batch_flowv="$(_flow_version)"; batch_grsha="$(_eval_gate_rules_sha)"
4337
+ fi
4219
4338
  echo "eval: batch $nonce (N=$n per fixture, timeout=${timeout}s)"
4220
4339
 
4221
4340
  # Ensure .flow/ (which now hosts eval-raw/ postmortem dumps) is git-ignored on any project
@@ -4246,7 +4365,9 @@ function cmd_eval {
4246
4365
  }
4247
4366
  END { print c + 0 }
4248
4367
  ' "$EVAL_MANIFEST")"
4249
- _eval_emit_batch_marker "$nonce" start "$n_expected"
4368
+ if [ "$replay_mode" -eq 0 ]; then
4369
+ _eval_emit_batch_marker "$nonce" start "$n_expected"
4370
+ fi
4250
4371
 
4251
4372
  # Scoped INT/TERM trap: the sole EXIT trap (_log_on_exit -> _cleanup_tds) was found NOT to
4252
4373
  # fire reliably while blocked on a foreground `_run_with_timeout` child (verified: a signal
@@ -4293,6 +4414,11 @@ function cmd_eval {
4293
4414
 
4294
4415
  local promptfile="$rundir/prompt.txt"
4295
4416
  if ! _eval_build_prompt "$promptfile" "$fstage_short" "$artifact_path" "$nonce"; then
4417
+ if [ "$replay_mode" -eq 1 ]; then
4418
+ echo " $fid: SKIP - no heading-mapped section (artifact-replay scope is 01|02|card)"
4419
+ rm -rf "$rundir" 2>/dev/null
4420
+ continue
4421
+ fi
4296
4422
  echo " $fid: FAIL - gate-rules.md section for stage '$fstage' extracted EMPTY (heading map or file drift) - aborted before any billable call"
4297
4423
  total_mismatch=$((total_mismatch + 1))
4298
4424
  rm -rf "$rundir" 2>/dev/null
@@ -4312,7 +4438,12 @@ function cmd_eval {
4312
4438
  # command string _eval_engine_run passes to _run_with_timeout (that broke the watchdog
4313
4439
  # fallback on the timeout-less-PATH lane - see engine-run's note).
4314
4440
  err1="$rundir/v${i}-a1.err"
4315
- raw="$(_eval_engine_run "$promptfile" "$timeout" 2>"$err1")"; rc=$?
4441
+ if [ "$replay_mode" -eq 1 ]; then
4442
+ raw="$(_eval_replay_vote "$fid" "$i")" || { rm -rf "$rundir" 2>/dev/null; trap - INT TERM; return 1; }
4443
+ rc=0
4444
+ else
4445
+ raw="$(_eval_engine_run "$promptfile" "$timeout" 2>"$err1")"; rc=$?
4446
+ fi
4316
4447
  raw1="$raw"; rc1="$rc"
4317
4448
  if [ "$rc" -eq 124 ]; then
4318
4449
  v=INVALID
@@ -4332,7 +4463,7 @@ function cmd_eval {
4332
4463
  # _run_with_timeout is documented not to bound a stuck call (DEBT.md), a retry here just
4333
4464
  # waits another full stuck-call duration - test E on the 3-OS CI matrix reproduces this at
4334
4465
  # ~66s (2x ~30s stuck calls + backoff) despite --timeout 3.
4335
- if [ "$v" = "INVALID" ] && [ "$rl1" != "true" ] && [ "$rc1" -ne 124 ]; then
4466
+ if [ "$replay_mode" -eq 0 ] && [ "$v" = "INVALID" ] && [ "$rl1" != "true" ] && [ "$rc1" -ne 124 ]; then
4336
4467
  # Backoff then retry - env-injectable so tests can zero it; skipped when rc=124 (timeout
4337
4468
  # was infra, retry would timeout again) or when rate-limited (see above).
4338
4469
  if [ "$backoff" -gt 0 ]; then
@@ -4342,7 +4473,12 @@ function cmd_eval {
4342
4473
  echo " $fid: retrying vote $i (parse-INVALID on attempt 1)"
4343
4474
  fi
4344
4475
  err2="$rundir/v${i}-a2.err"
4345
- raw="$(_eval_engine_run "$promptfile" "$timeout" 2>"$err2")"; rc=$?
4476
+ if [ "$replay_mode" -eq 1 ]; then
4477
+ raw="$(_eval_replay_vote "$fid" "$i")" || { rm -rf "$rundir" 2>/dev/null; trap - INT TERM; return 1; }
4478
+ rc=0
4479
+ else
4480
+ raw="$(_eval_engine_run "$promptfile" "$timeout" 2>"$err2")"; rc=$?
4481
+ fi
4346
4482
  retries_sum=$((retries_sum + 1))
4347
4483
  if [ "$rc" -eq 124 ]; then v=INVALID; else v="$(_eval_parse_verdict "$raw" "$nonce")"; fi
4348
4484
  [ "$model" = "unknown" ] && [ "$rc" -ne 124 ] && model="$(_eval_parse_model "$raw")"
@@ -4385,6 +4521,9 @@ function cmd_eval {
4385
4521
  PASS) pass_count=$((pass_count + 1)) ;;
4386
4522
  *) invalid_count=$((invalid_count + 1)) ;;
4387
4523
  esac
4524
+ if [ "$record_mode" -eq 1 ]; then
4525
+ _eval_record_vote "$fid" "$i" "$nonce" "$v"
4526
+ fi
4388
4527
  i=$((i + 1))
4389
4528
  done
4390
4529
  rm -rf "$rundir" 2>/dev/null
@@ -4415,7 +4554,9 @@ function cmd_eval {
4415
4554
  echo " $fid ($fstage): $verdict - MISMATCH, expected $fexpected (flag=$flag_count pass=$pass_count invalid=$invalid_count)"
4416
4555
  fi
4417
4556
 
4418
- _eval_emit_result "$nonce" "$fid" "$fstage" "$fexpected" "$verdict" "$match" "$flag_count" "$pass_count" "$invalid_count" "$n" "$model" "$batch_cliv" "$batch_flowv" "$batch_grsha" "$retries_sum" "$vote_rate_limited"
4557
+ if [ "$replay_mode" -eq 0 ]; then
4558
+ _eval_emit_result "$nonce" "$fid" "$fstage" "$fexpected" "$verdict" "$match" "$flag_count" "$pass_count" "$invalid_count" "$n" "$model" "$batch_cliv" "$batch_flowv" "$batch_grsha" "$retries_sum" "$vote_rate_limited"
4559
+ fi
4419
4560
  n_written=$((n_written + 1))
4420
4561
 
4421
4562
  # Circuit breaker: catches the 260710-class INVALID storm early. Trip when the FIRST
@@ -4442,9 +4583,12 @@ function cmd_eval {
4442
4583
  # SKIPS the trailer so an early-broken batch never satisfies --report's completeness check
4443
4584
  # (n_written>=n_expected AND trailer present), even on a --fixture-filtered run where
4444
4585
  # n_written could accidentally equal the filtered n_expected of 1.
4445
- if [ "$aborted" -eq 0 ]; then
4586
+ if [ "$aborted" -eq 0 ] && [ "$replay_mode" -eq 0 ]; then
4446
4587
  _eval_emit_batch_marker "$nonce" done "$n_written"
4447
4588
  fi
4589
+ if [ "$record_mode" -eq 1 ] && [ "$aborted" -eq 0 ]; then
4590
+ _eval_replay_write_meta "$nonce" "$batch_grsha" "${model:-unknown}" "$n"
4591
+ fi
4448
4592
 
4449
4593
  echo
4450
4594
  if [ "$total_evaluated" -eq 0 ]; then
@@ -4457,8 +4601,13 @@ function cmd_eval {
4457
4601
  echo "ABORTED: circuit breaker tripped on first fixture UNRELIABLE - $total_evaluated of $n_expected fixtures evaluated."
4458
4602
  return 2
4459
4603
  fi
4460
- _eval_print_scorecard "$nonce"
4604
+ if [ "$replay_mode" -eq 0 ]; then
4605
+ _eval_print_scorecard "$nonce"
4606
+ fi
4461
4607
  echo
4608
+ if [ "$replay_mode" -eq 1 ]; then
4609
+ echo "replay: $total_evaluated fixture(s) (keyless)"
4610
+ fi
4462
4611
  if [ "$total_mismatch" -eq 0 ] && [ "$total_unreliable" -eq 0 ]; then
4463
4612
  echo "PASS: all $total_evaluated evaluated fixture(s) majority-matched their expected verdict."
4464
4613
  return 0
@@ -4506,9 +4655,14 @@ usage: bash flow.sh <command> [args]
4506
4655
  promote <file> Copy a playbook into the cross-project KB (~/.claude/flow/playbooks)
4507
4656
  doctor Check the environment (bash/python/grep/git) across macOS/Linux/Windows
4508
4657
  eval [opts] Behavioral eval: does the LLM semantic gate flag a hollow-but-mechanically-clean
4509
- fixture? Opt-in, BILLABLE (skips cleanly with zero calls if 'claude' CLI absent).
4658
+ fixture? Opt-in, BILLABLE (live mode skips cleanly with zero calls if 'claude'
4659
+ CLI absent; --replay never inherits that SKIP).
4510
4660
  [--stage 01|02|05|card|routing|converge] [--fixture <id>] [--n 3] [--timeout <seconds>]
4511
4661
  [--report] offline: last complete batch's scorecard + drift, zero calls
4662
+ [--record] live: write stripped transcripts under eval/replay/ (billable)
4663
+ [--replay] keyless: feed recorded transcripts through parse/vote/scorecard
4664
+ --record|--replay are artifact-modality only (not --stage routing|converge,
4665
+ not --report). --replay is not a fresh-judge and never counts toward the floor.
4512
4666
  --stage routing: SEPARATE judge (concierge routing, not gate-rules FLAG/PASS);
4513
4667
  hard-capped at 90 calls/batch, own results stream, own --report
4514
4668
  retro Print the 3 retro questions
@@ -1,8 +1,8 @@
1
1
  {
2
2
  "source": "../skills/flow",
3
3
  "destRelative": "skills/flow",
4
- "fileCount": 100,
5
- "syncedAt": "2026-08-13T12:49:42.684Z",
4
+ "fileCount": 102,
5
+ "syncedAt": "2026-08-14T09:19:34.115Z",
6
6
  "fileList": [
7
7
  "SKILL.md",
8
8
  "_templates/00-idea.md",
@@ -35,6 +35,8 @@
35
35
  "eval/fixtures/fcdb/cards/C-001.md",
36
36
  "eval/fixtures/fcdc/.flow/.gitkeep",
37
37
  "eval/fixtures/fcdc/cards/C-001.md",
38
+ "eval/fixtures/fcdd/cards/C-001.md",
39
+ "eval/fixtures/fcde/cards/C-001.md",
38
40
  "eval/fixtures/routing/manifest.tsv",
39
41
  "eval/fixtures/routing/states/state-any-midproject.txt",
40
42
  "eval/fixtures/routing/states/state-cards-todo.txt",