@manhquy/flow-skill 0.6.0 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +23 -0
- package/README.md +6 -5
- package/README_VN.md +7 -6
- package/package.json +1 -1
- package/skills/flow/SKILL.md +4 -3
- package/skills/flow/eval/fixtures/fcdd/cards/C-001.md +34 -0
- package/skills/flow/eval/fixtures/fcde/cards/C-001.md +46 -0
- package/skills/flow/eval/manifest.tsv +2 -0
- package/skills/flow/references/command-dispatch.md +2 -1
- package/skills/flow/references/gate-eval.md +51 -9
- package/skills/flow/references/gate-rules.md +3 -0
- package/skills/flow/references/ground-truth-gates.md +7 -0
- package/skills/flow/runner/flow.sh +178 -24
- package/skills-manifest.json +4 -2
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,29 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to `@manhquy/flow-skill`. Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Versioning: [SemVer](https://semver.org/).
|
|
4
4
|
|
|
5
|
+
## [0.7.0] — 2026-08-14 — ships skill v0.30.0 (discipline-layer identity + CI/eval hardening)
|
|
6
|
+
|
|
7
|
+
Skill-content release. Ships skill product **v0.30.0**: identity ADR (gates and
|
|
8
|
+
receipts, never the runtime), keyless `eval --record`/`--replay`, pack-rehearsal,
|
|
9
|
+
`all-checks-passed`, macOS live-eval refuse-guard, B1-S `fcdd`/`fcde` (see root
|
|
10
|
+
`CHANGELOG.md` `## 0.30.0`). Installer CLI contract unchanged. Publishes to
|
|
11
|
+
dist-tag **`latest`**.
|
|
12
|
+
|
|
13
|
+
### Changed
|
|
14
|
+
- Bundled skill tree re-synced from monorepo `skills/flow/`.
|
|
15
|
+
- Install: **`npx @manhquy/flow-skill@latest`**.
|
|
16
|
+
|
|
17
|
+
## [0.6.0] — 2026-08-13 — ships skill v0.29.0 (spec-kit imports + converge)
|
|
18
|
+
|
|
19
|
+
Skill-content release. Ships skill product **v0.29.0**: `/flow clarify`, card
|
|
20
|
+
independent-test field, `/flow converge`, converge eval modality (see root
|
|
21
|
+
`CHANGELOG.md` `## 0.29.0`). Installer CLI contract unchanged. Publishes to
|
|
22
|
+
dist-tag **`latest`**.
|
|
23
|
+
|
|
24
|
+
### Changed
|
|
25
|
+
- Bundled skill tree re-synced from monorepo `skills/flow/`.
|
|
26
|
+
- Install: **`npx @manhquy/flow-skill@latest`**.
|
|
27
|
+
|
|
5
28
|
## [0.5.0] — 2026-08-11 — ships skill v0.28.0 (attested execution)
|
|
6
29
|
|
|
7
30
|
Installer minor: bundles skill **v0.28.0** (risk + attestation receipts + auto trust plane).
|
package/README.md
CHANGED
|
@@ -5,10 +5,11 @@ Install the **flow** skill into coding agents with one command.
|
|
|
5
5
|
| | |
|
|
6
6
|
|---|---|
|
|
7
7
|
| **Package** | [`@manhquy/flow-skill`](https://www.npmjs.com/package/@manhquy/flow-skill) (installer CLI) |
|
|
8
|
-
| **Current GA** | **0.
|
|
8
|
+
| **Current GA** | **0.7.0** — ships skill product **v0.30.0** |
|
|
9
9
|
| **Platforms** | macOS · Linux · Windows (same Node path) |
|
|
10
10
|
| **Requirement** | Node.js **≥ 22.14** |
|
|
11
11
|
| **Source** | [github.com/manhquydev/flow-skill](https://github.com/manhquydev/flow-skill) |
|
|
12
|
+
| **Website** | [flowskill.io.vn](https://flowskill.io.vn) |
|
|
12
13
|
|
|
13
14
|
flow is a gated build harness (`/flow`): idea → honest gates → real done-evidence. This package only **installs** that skill into agent skill homes.
|
|
14
15
|
|
|
@@ -34,7 +35,7 @@ npx @manhquy/flow-skill@latest
|
|
|
34
35
|
|
|
35
36
|
```bash
|
|
36
37
|
npx @manhquy/flow-skill@latest --help
|
|
37
|
-
# expect: flow-skill v0.
|
|
38
|
+
# expect: flow-skill v0.7.x (ships skill v0.30.x)
|
|
38
39
|
```
|
|
39
40
|
|
|
40
41
|
### Common commands
|
|
@@ -89,7 +90,7 @@ npx @manhquy/flow-skill@latest --yes --all --dry-run --json
|
|
|
89
90
|
|----|--------|
|
|
90
91
|
| `npx @manhquy/flow-skill@latest` | Bare `npx @manhquy/flow-skill` (stale npx cache) |
|
|
91
92
|
| **Run** the CLI to copy the skill | `npm i` alone (package only; no skill files in agent homes) |
|
|
92
|
-
| Pin installer `@0.
|
|
93
|
+
| Pin installer `@0.7.0` if needed | Pin npm `@0.30.0` (skill version ≠ package version) |
|
|
93
94
|
| Prefer `@latest` | `@rc` (retired / behind) |
|
|
94
95
|
|
|
95
96
|
**Two version axes:** `package.json` version = installer; `SKILL.md` metadata = skill product. Both appear in `--help` and in the `plan` JSONL event (`version` + `skillVersion`).
|
|
@@ -137,7 +138,7 @@ rm -rf <project>/.claude/skills/flow
|
|
|
137
138
|
|---------|-----|
|
|
138
139
|
| Agent has no `/flow` after `npm i` | Run `npx @manhquy/flow-skill@latest` (install must **execute**) |
|
|
139
140
|
| Old skill after reinstall | Always use `@latest`; avoid bare package name |
|
|
140
|
-
| `No matching version found` for `@0.
|
|
141
|
+
| `No matching version found` for `@0.30.0` | That is the **skill** version. Use `@latest` or `@0.7.x` |
|
|
141
142
|
| Node too old | Install Node ≥ 22.14 (`nvm install 22`, etc.) |
|
|
142
143
|
| Windows `EBUSY` / `EPERM` | Close the agent holding files under the skill dir; re-run |
|
|
143
144
|
| Stale install lock | Delete `<parent>/.flow-skill.installing.lock` if reclaim fails |
|
|
@@ -147,7 +148,7 @@ rm -rf <project>/.claude/skills/flow
|
|
|
147
148
|
## JSONL (`--json`)
|
|
148
149
|
|
|
149
150
|
```jsonl
|
|
150
|
-
{"event":"plan","version":"0.
|
|
151
|
+
{"event":"plan","version":"0.7.0","skillVersion":"0.30.0","dryRun":false,"scope":"global","targets":["claude"]}
|
|
151
152
|
{"event":"install:start","target":"claude","dests":["~/.claude/skills/flow"]}
|
|
152
153
|
{"event":"install:done","target":"claude","dests":["~/.claude/skills/flow"],"result":"success","error":null,"warnings":[]}
|
|
153
154
|
{"event":"summary","success":true,"total":1,"attempted":1,"installed":1,"failed":0,"skipped":0,"aborted":false}
|
package/README_VN.md
CHANGED
|
@@ -5,10 +5,11 @@ Cài skill **flow** vào coding agent bằng một lệnh.
|
|
|
5
5
|
| | |
|
|
6
6
|
|---|---|
|
|
7
7
|
| **Package** | [`@manhquy/flow-skill`](https://www.npmjs.com/package/@manhquy/flow-skill) (installer CLI) |
|
|
8
|
-
| **GA hiện tại** | **0.
|
|
8
|
+
| **GA hiện tại** | **0.7.0** — ships skill product **v0.30.0** |
|
|
9
9
|
| **Nền tảng** | macOS · Linux · Windows (cùng code path Node) |
|
|
10
10
|
| **Yêu cầu** | Node.js **≥ 22.14** |
|
|
11
11
|
| **Source** | [github.com/manhquydev/flow-skill](https://github.com/manhquydev/flow-skill) |
|
|
12
|
+
| **Website** | [flowskill.io.vn](https://flowskill.io.vn) |
|
|
12
13
|
|
|
13
14
|
flow là harness build có cổng (`/flow`): ý tưởng → cổng trung thực → bằng chứng done thật. Package này chỉ **cài** skill đó vào skill home của agent.
|
|
14
15
|
|
|
@@ -34,7 +35,7 @@ npx @manhquy/flow-skill@latest
|
|
|
34
35
|
|
|
35
36
|
```bash
|
|
36
37
|
npx @manhquy/flow-skill@latest --help
|
|
37
|
-
# expect: flow-skill v0.
|
|
38
|
+
# expect: flow-skill v0.7.x (ships skill v0.30.x)
|
|
38
39
|
```
|
|
39
40
|
|
|
40
41
|
### Lệnh hay dùng
|
|
@@ -89,8 +90,8 @@ npx @manhquy/flow-skill@latest --yes --all --dry-run --json
|
|
|
89
90
|
|-----|--------|
|
|
90
91
|
| `npx @manhquy/flow-skill@latest` | Bare `npx @manhquy/flow-skill` (cache npx có thể cũ) |
|
|
91
92
|
| **Chạy** CLI để copy skill | Chỉ `npm i` (chỉ thêm package; không cài skill vào agent home) |
|
|
92
|
-
| Pin installer `@0.
|
|
93
|
-
| Ưu tiên `@latest` / `@0.
|
|
93
|
+
| Pin installer `@0.7.0` nếu cần cố định | Pin npm `@0.30.0` (version **skill** ≠ version package) |
|
|
94
|
+
| Ưu tiên `@latest` / `@0.7.x` | `@rc` (đã retire / tụt hậu) |
|
|
94
95
|
|
|
95
96
|
**Hai trục version:** `package.json` = installer CLI; `SKILL.md` metadata = skill product. Cả hai hiện trong `--help` và event JSONL `plan` (`version` + `skillVersion`).
|
|
96
97
|
|
|
@@ -137,7 +138,7 @@ rm -rf <project>/.claude/skills/flow
|
|
|
137
138
|
|------------|------------|
|
|
138
139
|
| Agent không có `/flow` sau `npm i` | Chạy `npx @manhquy/flow-skill@latest` (phải **execute** installer) |
|
|
139
140
|
| Skill cũ sau khi “cài lại” | Luôn dùng `@latest`; tránh bare package name |
|
|
140
|
-
| `No matching version found` với `@0.
|
|
141
|
+
| `No matching version found` với `@0.30.0` | Đó là version **skill**. Dùng `@latest` hoặc `@0.7.x` |
|
|
141
142
|
| Node quá cũ | Cài Node ≥ 22.14 (`nvm install 22`, v.v.) |
|
|
142
143
|
| Windows `EBUSY` / `EPERM` | Đóng agent đang giữ file trong skill dir; chạy lại |
|
|
143
144
|
| Stale install lock | Xóa `<parent>/.flow-skill.installing.lock` nếu reclaim thất bại |
|
|
@@ -147,7 +148,7 @@ rm -rf <project>/.claude/skills/flow
|
|
|
147
148
|
## JSONL (`--json`)
|
|
148
149
|
|
|
149
150
|
```jsonl
|
|
150
|
-
{"event":"plan","version":"0.
|
|
151
|
+
{"event":"plan","version":"0.7.0","skillVersion":"0.30.0","dryRun":false,"scope":"global","targets":["claude"]}
|
|
151
152
|
{"event":"install:start","target":"claude","dests":["~/.claude/skills/flow"]}
|
|
152
153
|
{"event":"install:done","target":"claude","dests":["~/.claude/skills/flow"],"result":"success","error":null,"warnings":[]}
|
|
153
154
|
{"event":"summary","success":true,"total":1,"attempted":1,"installed":1,"failed":0,"skipped":0,"aborted":false}
|
package/package.json
CHANGED
package/skills/flow/SKILL.md
CHANGED
|
@@ -3,12 +3,12 @@ name: flow
|
|
|
3
3
|
description: Run the buildflow gated build process from idea to real done-evidence. Walk gated stages (Idea->Research->Scope->PRD->ADR->Contract->Cards->Build->Review->Deploy/Ship->Verify->Retro), each with a honest gate that must pass before advancing. Adapts to project type (web|cli|library|skill). Use when starting or driving a real product build, when the user types /flow, /flow next, /flow card, /flow check, or asks to scope/plan/ship a project through gates. Kill at any gate is a valid outcome.
|
|
4
4
|
user-invocable: true
|
|
5
5
|
when_to_use: "User wants to build a real product end-to-end with discipline (idea -> a deployed URL for web, or installs+runs for a CLI/library/skill), or types any /flow command, or asks for a gated build process, scope decision, contract-first plan, or card-based shipping."
|
|
6
|
-
argument-hint: "[ resume | next | card | check C-NNN | project-type web|cli|library|skill | mode teach|work | skip <stage> | ready | workspace add|list|enter|remove|check|doctor | auto [stop] | attest semantic|live-verify|status|recover | doctor | retro | eval --stage 01|02|05|card|routing|converge --fixture <id> --n 3 --keep-going --report | clarify | converge | or just say what you want in plain language ]"
|
|
6
|
+
argument-hint: "[ resume | next | card | check C-NNN | project-type web|cli|library|skill | mode teach|work | skip <stage> | ready | workspace add|list|enter|remove|check|doctor | auto [stop] | attest semantic|live-verify|status|recover | doctor | retro | eval --stage 01|02|05|card|routing|converge --fixture <id> --n 3 --keep-going --report --record|--replay | clarify | converge | or just say what you want in plain language ]"
|
|
7
7
|
keywords: [flow, buildflow, gate, build, ship, scope, prd, contract, card, deploy, vertical-slice, cli, library, skill, worktree, parallel-agents, workspace, multi-agent]
|
|
8
8
|
license: MIT
|
|
9
9
|
metadata:
|
|
10
10
|
author: flow-skill
|
|
11
|
-
version: "0.
|
|
11
|
+
version: "0.30.0"
|
|
12
12
|
attribution: "Methodology from ai20k-build-phase/buildflow (Tony, arealisticdreamer.com); durable-layer ancestry and improve-ritual spirit from repository-harness (protocol v1 EOL — flow-owned fork); agent patterns from claudekit-engineer, BMAD-METHOD. v0.22 concierge from BMAD bmad-help; forge-idea from bmad-forge-idea (MIT, BMad Code LLC)."
|
|
13
13
|
---
|
|
14
14
|
|
|
@@ -107,8 +107,9 @@ takes over a lock you're sure is dead; `/flow unlock` clears it.
|
|
|
107
107
|
| `/flow constitution` | `flow.sh constitution` — check operator-authored per-project invariants in `flow/constitution.md` (structure + optional grep-markers); **advisory and NOT a `next` gate** — run it at the scope/PRD/contract seam, then apply the semantic challenge in `gate-rules.md` |
|
|
108
108
|
| `/flow clarify` | `flow.sh clarify` — list leftover `- [ ]` bullets under `## Open decisions` on scope/PRD/contract (section-scoped, always exit 0); **advisory and NOT a `next` gate**. Write-back is the ritual in `references/clarify.md` (opt-in). |
|
|
109
109
|
| `/flow converge` | `flow.sh converge [--file <payload>]` — **append-only** remainder cards reconciling present code vs plan. Assess per `references/converge.md`, write a `flow-converge/v1` payload, then run: transactional (all cards or none), never edits an existing card, prints `CONVERGED` + writes nothing when there is no gap. Semantic proof: `flow.sh eval --stage converge`. |
|
|
110
|
-
| `/flow eval [--stage 01\|02\|card] [--fixture <id>] [--n 3] [--timeout <s>] [--keep-going]` | `flow.sh eval` — **behavioral eval**: does the LLM semantic gate (`gate-rules.md`) actually flag a hollow-but-mechanically-clean fixture? Opt-in, **billable** (skips cleanly, zero calls, if `claude` CLI absent); prints a per-stage scorecard. v0.21: on a final-INVALID vote, raw stdout + stderr + rc are persisted to `.flow/eval-raw/<run_id>/` (git-ignored, envelope-stripped) so an INVALID storm is postmortemable; the first UNRELIABLE fixture aborts the batch (`--keep-going` forces full-batch, worst-case ~37 calls at `--n 3`); the retry backoff is env-injectable (`FLOW_EVAL_RETRY_BACKOFF`, default 5s, set 0 in tests). See `references/gate-eval.md` for scope/cost/limitations. |
|
|
110
|
+
| `/flow eval [--stage 01\|02\|card] [--fixture <id>] [--n 3] [--timeout <s>] [--keep-going]` | `flow.sh eval` — **behavioral eval**: does the LLM semantic gate (`gate-rules.md`) actually flag a hollow-but-mechanically-clean fixture? Opt-in, **billable** (live mode skips cleanly, zero calls, if `claude` CLI absent; `--replay` never inherits that SKIP); prints a per-stage scorecard. v0.21: on a final-INVALID vote, raw stdout + stderr + rc are persisted to `.flow/eval-raw/<run_id>/` (git-ignored, envelope-stripped) so an INVALID storm is postmortemable; the first UNRELIABLE fixture aborts the batch (`--keep-going` forces full-batch, worst-case ~37 calls at `--n 3`); the retry backoff is env-injectable (`FLOW_EVAL_RETRY_BACKOFF`, default 5s, set 0 in tests). See `references/gate-eval.md` for scope/cost/limitations. |
|
|
111
111
|
| `/flow eval --report` | `flow.sh eval --report` — **offline**, zero calls: last complete batch's scorecard + drift vs the prior complete batch |
|
|
112
|
+
| `/flow eval --replay` | `flow.sh eval --replay` — **keyless**: replay recorded stripped transcripts through parse → vote → per-fixture match lines (no scorecard; replay writes no results rows). Not a fresh-judge; verdicts never count toward the eval floor. Stale gate-rules hash → exit 1 (re-record live with `--record`). |
|
|
112
113
|
| `/flow promote <file>` | `flow.sh promote <file>` — copy a playbook into the cross-project KB (`~/.claude/flow/playbooks`); `recall` then surfaces it everywhere |
|
|
113
114
|
| `/flow project-type <web\|cli\|library\|skill>` | `flow.sh project-type` — set/read the project type that selects the per-type gate lens (`references/project-types.md`) |
|
|
114
115
|
| `/flow skip <stage>` | `flow.sh skip` — advance past a gate that has a matching open `DEBT.md` line; **security-class skips are operator-only and HALT** (never auto-skipped) |
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# C-001 — Add debt-simplification nudge to the household expense app
|
|
2
|
+
|
|
3
|
+
status: done
|
|
4
|
+
deps: none
|
|
5
|
+
implements: FR3
|
|
6
|
+
|
|
7
|
+
## Scope
|
|
8
|
+
|
|
9
|
+
Compute the minimum set of repayment transactions across a household group (N members,
|
|
10
|
+
arbitrary unequal splits) and surface it as a single "who pays whom, how much" nudge on
|
|
11
|
+
the group's balance screen.
|
|
12
|
+
|
|
13
|
+
## Allowed files
|
|
14
|
+
|
|
15
|
+
src/balance/simplify.ts, src/balance/simplify.test.ts, src/screens/GroupBalance.tsx
|
|
16
|
+
|
|
17
|
+
## Verify (run these before calling the card done)
|
|
18
|
+
|
|
19
|
+
- [x] Simplify cases for 2, 3, and 6-member groups all came back PASS
|
|
20
|
+
- [x] A 4-person demo group now settles in three transfers instead of the naive six
|
|
21
|
+
- [x] Nudge copy reviewed against the design law — no emoji, plain currency formatting
|
|
22
|
+
|
|
23
|
+
## Done-evidence (world-state proof, named BEFORE building)
|
|
24
|
+
|
|
25
|
+
A live staging surface where a 4-person group's balance screen shows the simplified nudge
|
|
26
|
+
("Alice owes Bob $12.50") instead of a raw pairwise list.
|
|
27
|
+
|
|
28
|
+
## Evidence (paste the actual proof here when done)
|
|
29
|
+
|
|
30
|
+
The 4-person demo group now settles in three transfers instead of the naive six. Alice
|
|
31
|
+
covers Bob and Carol covers Dana, which is the nudge the card asked for. A teammate
|
|
32
|
+
walked the same group after dinner and confirmed the amounts match what people actually
|
|
33
|
+
owed. The group ledger shows rows=4 after the split, and the simplify cases came back
|
|
34
|
+
PASS. Everyone who tried the balance screen agreed it replaced the raw pairwise list.
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
# C-001 — Add debt-simplification nudge to the household expense app
|
|
2
|
+
|
|
3
|
+
status: done
|
|
4
|
+
deps: none
|
|
5
|
+
implements: FR3
|
|
6
|
+
|
|
7
|
+
## Scope
|
|
8
|
+
|
|
9
|
+
Compute the minimum set of repayment transactions across a household group (N members,
|
|
10
|
+
arbitrary unequal splits) and surface it as a single "who pays whom, how much" nudge on
|
|
11
|
+
the group's balance screen.
|
|
12
|
+
|
|
13
|
+
## Allowed files
|
|
14
|
+
|
|
15
|
+
src/balance/simplify.ts, src/balance/simplify.test.ts, src/screens/GroupBalance.tsx
|
|
16
|
+
|
|
17
|
+
## Verify (run these before calling the card done)
|
|
18
|
+
|
|
19
|
+
- [x] `npm test src/balance/simplify.test.ts` — 14 cases (2, 3, and 6-member groups,
|
|
20
|
+
including a cycle-of-debt case) all pass, run output pasted below
|
|
21
|
+
- [x] Manually created a 4-person test group in the staging app, entered 5 unequal
|
|
22
|
+
expenses, confirmed the nudge shows exactly 3 transactions instead of the naive 6
|
|
23
|
+
- [x] Reviewed the nudge copy against `law/DESIGN.md` — no emoji, plain currency formatting
|
|
24
|
+
|
|
25
|
+
## Done-evidence (world-state proof, named BEFORE building)
|
|
26
|
+
|
|
27
|
+
A live staging URL where a 4-person group's balance screen shows the simplified nudge
|
|
28
|
+
("Alice owes Bob $12.50") instead of a raw pairwise list, screenshotted and linked below.
|
|
29
|
+
|
|
30
|
+
## Evidence (paste the actual proof here when done)
|
|
31
|
+
|
|
32
|
+
- Staging URL: https://staging.householdsplit.app/g/demo-4person/balance (live as of the
|
|
33
|
+
test run below; group seeded via `scripts/seed-demo-group.ts`)
|
|
34
|
+
- Test run output of `npm test src/balance/simplify.test.ts`:
|
|
35
|
+
```
|
|
36
|
+
PASS src/balance/simplify.test.ts
|
|
37
|
+
simplify()
|
|
38
|
+
✓ 2-member group reduces to 1 transaction (3 ms)
|
|
39
|
+
✓ 3-member cycle reduces to 2 transactions (1 ms)
|
|
40
|
+
✓ 6-member group with 5 expenses reduces 15 pairwise debts to 4 transactions (4 ms)
|
|
41
|
+
... 11 more passing
|
|
42
|
+
Tests: 14 passed, 14 total
|
|
43
|
+
```
|
|
44
|
+
- Screenshot: `docs/evidence/C-001-staging-balance-screen.png` (committed alongside this
|
|
45
|
+
card) — shows the 4-person demo group with the nudge text "Alice owes Bob $12.50" and
|
|
46
|
+
"Carol owes Dana $8.00" replacing the previous 6-line raw balance table.
|
|
@@ -6,5 +6,7 @@ f02b 02-scope flow/02-scope.md FLAG
|
|
|
6
6
|
fcda card cards/C-001.md PASS
|
|
7
7
|
fcdb card cards/C-001.md FLAG
|
|
8
8
|
fcdc card cards/C-001.md FLAG
|
|
9
|
+
fcdd card cards/C-001.md FLAG
|
|
10
|
+
fcde card cards/C-001.md PASS
|
|
9
11
|
f05a 05-contract flow/05-contract.md PASS
|
|
10
12
|
f05b 05-contract flow/05-contract.md FLAG
|
|
@@ -22,8 +22,9 @@ install dir; run from the project root so `flow/` and `cards/` resolve.
|
|
|
22
22
|
| `/flow constitution` | `bash <skill>/runner/flow.sh constitution` | At the scope/PRD/contract seam: checks `flow/constitution.md` is well-formed (no placeholder, every invariant has an ID) + scans declared grep-markers. **Advisory, NOT a `next` gate** — then apply the semantic challenge in `gate-rules.md`. |
|
|
23
23
|
| `/flow clarify` | `bash <skill>/runner/flow.sh clarify` | List leftover `- [ ]` bullets under `## Open decisions` on 02/03/05 (section-scoped). **Advisory, NOT a `next` gate** — write-back is `references/clarify.md` (opt-in). Never invent auth/tenancy/retention/billing to clear FILL. |
|
|
24
24
|
| `/flow converge` | `bash <skill>/runner/flow.sh converge [--file <payload>]` | Flow-back closer: assess present code vs plan per `references/converge.md`, write a `flow-converge/v1` payload to `.flow/converge-pending.md`, then run to **append** remainder cards. Transactional (all or none), append-only (never edits an existing card), prints `CONVERGED` + writes nothing when there is no gap. Present the findings table to the operator first. |
|
|
25
|
-
| `/flow eval [--stage 01\|02\|card] [--fixture <id>] [--n 3] [--timeout <s>]` | `bash <skill>/runner/flow.sh eval [...]` | **Billable** (skips cleanly, zero calls, if `claude` CLI absent). Relay the scorecard verbatim. See `references/gate-eval.md` before running: what it measures (a fresh-judge lower bound, not the work-mode self-challenge), cost, and thresholds. Never wired into `next`/`check` gating — informative only. |
|
|
25
|
+
| `/flow eval [--stage 01\|02\|card] [--fixture <id>] [--n 3] [--timeout <s>]` | `bash <skill>/runner/flow.sh eval [...]` | **Billable** (live mode skips cleanly, zero calls, if `claude` CLI absent). `--replay` does **not** inherit that SKIP — missing fixtures exit 1. Relay the scorecard verbatim. See `references/gate-eval.md` before running: what it measures (a fresh-judge lower bound, not the work-mode self-challenge), cost, and thresholds. Never wired into `next`/`check` gating — informative only. |
|
|
26
26
|
| `/flow eval --report` | `bash <skill>/runner/flow.sh eval --report` | Offline, zero calls. Relay the last complete batch's scorecard + drift vs the prior complete batch. |
|
|
27
|
+
| `/flow eval --replay` | `bash <skill>/runner/flow.sh eval --replay` | Keyless. Replay recorded stripped transcripts through parse → vote → per-fixture match lines (no scorecard; replay writes no results rows). Not a fresh-judge; verdicts never count toward the eval floor. Stale `gate_rules_sha` → exit 1 (re-record live). |
|
|
27
28
|
| `/flow project-type <web\|cli\|library\|skill>` | `bash <skill>/runner/flow.sh project-type [t]` | Set/read the project type that selects the per-type gate lens + done-evidence definition (`references/project-types.md`). Confirm the type matches reality before planning. |
|
|
28
29
|
| `/flow usage [--global\|--prune]` | `bash <skill>/runner/flow.sh usage [...]` | Roll up the JSONL usage log and relay the analytics (cycle-time, gate fail-rate, per-stage + per-card dwell, command breakdown). Local-only; `--prune [--keep N]` caps the log. Read-only — nothing to author. |
|
|
29
30
|
| `/flow skip <stage> --reason ...` | `bash <skill>/runner/flow.sh skip <stage> --reason ...` | Advance past a gate that has a matching open `DEBT.md` line. **Security-class skips are operator-only and HALT** — never auto-skip; the operator accepts the exposure in writing first. |
|
|
@@ -25,14 +25,18 @@ rule text is enforceable by *a* judge," not "this build's self-review caught eve
|
|
|
25
25
|
|
|
26
26
|
## Fixtures and thresholds
|
|
27
27
|
|
|
28
|
-
|
|
28
|
+
Eleven shipped fixtures (`skills/flow/eval/fixtures/`, manifest `skills/flow/eval/manifest.tsv`):
|
|
29
29
|
sound/hollow pairs for Stage 01 (Research — fabricated quotes, vague "users"), Stage 02 (Scope —
|
|
30
|
-
grade laundering, a real C quietly graded B),
|
|
30
|
+
grade laundering, a real C quietly graded B), Stage 05 (listed; heading map does not yet judge
|
|
31
|
+
them), and the card gate:
|
|
31
32
|
- **fcda** — sound multi-signal Evidence (mechanical PASS, semantic PASS)
|
|
32
33
|
- **fcdb** — process-only hollow (mechanical **FAIL** after hollow-done floor; still listed for
|
|
33
34
|
historical FLAG text / offline inspection)
|
|
34
35
|
- **fcdc** — hollow-with-decoy multi-signal (mechanical PASS, semantic **FLAG** — the residual
|
|
35
36
|
attack class the mechanical floor does not claim to close)
|
|
37
|
+
- **fcdd** — B1-S artifact-less Evidence prose (mechanical PASS, semantic **FLAG** — names no
|
|
38
|
+
path/URL/command; the semantic-only catch for `ground-truth-gates.md` rule 8)
|
|
39
|
+
- **fcde** — B1-S same story with concrete artifact refs (mechanical PASS, semantic PASS)
|
|
36
40
|
|
|
37
41
|
Fixture content maps 1:1 to the failure modes already documented in `gate-rules.md`, not an
|
|
38
42
|
invented taxonomy. The v1 input set is **manifest-listed shipped fixtures only** — `eval` never
|
|
@@ -61,8 +65,10 @@ separately from a real mismatch — never silently counted as a gate pass or fai
|
|
|
61
65
|
`eval` is **opt-in and billable** — it makes real `claude -p` API calls. Zero cost when the
|
|
62
66
|
`claude` CLI isn't on `PATH` (clean skip, exit 0). When present, exactly one minimal probe call
|
|
63
67
|
is made to confirm the CLI runs headless before any real judging starts; a probe failure means
|
|
64
|
-
one billable call was made, then a clean skip.
|
|
65
|
-
|
|
68
|
+
one billable call was made, then a clean skip. The artifact manifest is 11 rows; a full
|
|
69
|
+
default batch judges the 9 heading-mapped fixtures (01 / 02 / card) × N=3 = 27 judge calls
|
|
70
|
+
+ the 1 probe = 28 calls. The two Stage 05 rows fail closed before a call. Plan accounting
|
|
71
|
+
for a complete re-record is 11 × 3 = 33 + probe.
|
|
66
72
|
|
|
67
73
|
**Measured real cost** (this machine, 2026-07-10, default/unforced model, no `--bare` available
|
|
68
74
|
under an OAuth/subscription session — see the build's spike notes): roughly **$0.30–0.37 per
|
|
@@ -78,6 +84,39 @@ fresh-judge eval is to reflect a realistic judge, not the cheapest one that pass
|
|
|
78
84
|
`--report` (below) is the free, offline way to re-read a prior batch's numbers without spending
|
|
79
85
|
anything.
|
|
80
86
|
|
|
87
|
+
**macOS without `timeout`/`gtimeout`:** live `flow.sh eval` (not `--report`, not `--replay`) refuses
|
|
88
|
+
by default. The watchdog fallback cannot bound a stuck `claude` call, so a hung judge is an
|
|
89
|
+
unbounded-billing risk. Install GNU coreutils (`gtimeout`) or export `FLOW_EVAL_UNBOUNDED=1` to
|
|
90
|
+
accept that risk for one run. `--report` and `--replay` never hit this guard. `--record` is live
|
|
91
|
+
and does.
|
|
92
|
+
|
|
93
|
+
## Modes
|
|
94
|
+
|
|
95
|
+
| Mode | Calls | What it proves |
|
|
96
|
+
|------|-------|----------------|
|
|
97
|
+
| live (default) | Billable `claude -p` | Fresh-judge catch-rate (eval floor) |
|
|
98
|
+
| `--report` | Zero | Last complete **live** batch + advisory drift |
|
|
99
|
+
| `--record` | Same as live | Writes stripped transcripts under `eval/replay/` |
|
|
100
|
+
| `--replay` | Zero | Parse → vote → per-fixture match lines (no scorecard; no results rows) + gate-rules **hash** staleness |
|
|
101
|
+
|
|
102
|
+
`--record|--replay` are artifact-modality only (not `--stage routing|converge`, not `--report`).
|
|
103
|
+
|
|
104
|
+
**Replay is not a fresh-judge.** Gate-rules text is embedded in the live prompt; recorded
|
|
105
|
+
responses are frozen. Replay cannot test rules *effectiveness* — that stays live and billable.
|
|
106
|
+
What it can do: hard-fail when `_eval_gate_rules_sha` ≠ the recorded hash (`fixtures stale —
|
|
107
|
+
re-record live per ADR re-baseline rule`). Replay verdicts **never count toward the eval floor**.
|
|
108
|
+
|
|
109
|
+
**Refresh protocol:** edit both sides of a rules change, run a live `--record` batch (11 × `--n 3`
|
|
110
|
+
= 33 judge calls + 1 probe; 9 heading-mapped fixtures bill 27 + probe today) on a host with
|
|
111
|
+
real `timeout`/`gtimeout`, commit the stripped `eval/replay/` tree (verdict lines only — no
|
|
112
|
+
`session_id` / `cwd` envelopes) **in the same change** as the `gate-rules.md` edit. No
|
|
113
|
+
operator-recorded batch exists yet — the `eval-replay` CI job skip-with-notice until then;
|
|
114
|
+
the recorded-sha staleness check is unchanged.
|
|
115
|
+
|
|
116
|
+
**B1 escalation:** if hollow-done decoys that name no artifact/command recur in dogfood or
|
|
117
|
+
live eval after the B1-S addendum lands, escalate to full structured lineage evidence. Replay
|
|
118
|
+
verdicts never count toward that trigger.
|
|
119
|
+
|
|
81
120
|
## Running it
|
|
82
121
|
|
|
83
122
|
```
|
|
@@ -86,6 +125,8 @@ flow.sh eval --stage card # only the card-gate fixtures
|
|
|
86
125
|
flow.sh eval --fixture fcdb --n 1 # one fixture, one run (cheap smoke check)
|
|
87
126
|
flow.sh eval --timeout 180 # override the per-call timeout (default 120s)
|
|
88
127
|
flow.sh eval --report # OFFLINE: last complete batch's scorecard + drift
|
|
128
|
+
flow.sh eval --record --n 3 # LIVE: write stripped replay fixtures (billable)
|
|
129
|
+
flow.sh eval --replay --n 3 # KEYLESS: replay recorded transcripts
|
|
89
130
|
```
|
|
90
131
|
|
|
91
132
|
Exit code: `0` = every evaluated fixture majority-matched its expected verdict (or a clean
|
|
@@ -146,9 +187,10 @@ yet (the same "measure, don't assume" discipline that gated this build in the fi
|
|
|
146
187
|
platform: a signal arriving while the runner is blocked on a foreground judge call was not
|
|
147
188
|
observed to preempt that call promptly. Cleanup between fixtures/calls fires reliably; a
|
|
148
189
|
normal (uninterrupted) run always cleans up its temp files immediately.
|
|
149
|
-
- **Small fixture corpus (
|
|
150
|
-
enough for statistical confidence across every
|
|
151
|
-
corpus is a natural v1.x follow-up, not a
|
|
190
|
+
- **Small fixture corpus (eleven manifest rows, nine heading-mapped)**: enough to prove the
|
|
191
|
+
mechanism and give a real number, not enough for statistical confidence across every
|
|
192
|
+
stage/failure-mode combination. Widening the corpus is a natural v1.x follow-up, not a
|
|
193
|
+
v1 blocker.
|
|
152
194
|
|
|
153
195
|
## Failure modes and postmortem (v0.21)
|
|
154
196
|
|
|
@@ -174,8 +216,8 @@ window; the mechanism was never confirmed at the time (rate-limit vs hook conten
|
|
|
174
216
|
4. **Circuit breaker.** The FIRST evaluated fixture coming back UNRELIABLE (`invalid_count*3 > n`)
|
|
175
217
|
aborts the batch with a distinct nonzero exit code (2) and NO `done` trailer written — the
|
|
176
218
|
junk batch is invisible to `--report`/drift, so it cannot poison the baseline. `--keep-going`
|
|
177
|
-
forces the full batch; document worst case (≤ N_fixtures × n × 2 + 1 probe ≈
|
|
178
|
-
default
|
|
219
|
+
forces the full batch; document worst case (≤ N_fixtures × n × 2 + 1 probe ≈ 55 calls at
|
|
220
|
+
default 9 heading-mapped × 3).
|
|
179
221
|
|
|
180
222
|
### Rate-limit visibility (advisory, best-effort)
|
|
181
223
|
|
|
@@ -182,6 +182,9 @@ evidence non-empty.
|
|
|
182
182
|
- **Is `## Evidence` real world-state** — a clickable URL, real curl output, a DB row — and
|
|
183
183
|
NOT "tests pass" / "code merged" / "deployed successfully"? Merge != shipped: the proof is
|
|
184
184
|
the live surface changing, verified as a user.
|
|
185
|
+
- **Does every `## Evidence` item name its artifact (path/URL) or the command that produced
|
|
186
|
+
it?** Plausible prose that names neither is still hollow even when the mechanical floor
|
|
187
|
+
passes (`ground-truth-gates.md` rule 8).
|
|
185
188
|
- **If this card fixes a bug/regression**, the evidence includes the red→green proof that the
|
|
186
189
|
new test was actually tied to the bug (`ground-truth-gates.md` → "Bug-fix cards"), not just a
|
|
187
190
|
green run.
|
|
@@ -40,6 +40,13 @@ and used only to add color, never to pass a gate.
|
|
|
40
40
|
7. **Hand-edit `status: done` is not trusted for deps** until Evidence **and** Verify boxes
|
|
41
41
|
pass the same floor (`ready` re-validates Evidence + unchecked Verify + `[FILL]`; graph
|
|
42
42
|
executor mirrors).
|
|
43
|
+
8. **Every done-evidence item names its artifact or command.** Each `## Evidence` item must
|
|
44
|
+
name the path/URL of the artifact or the command that produced it. Plausible prose that
|
|
45
|
+
names neither is hollow even when the mechanical multi-signal floor passes — the card
|
|
46
|
+
gate's semantic challenge catches it (`gate-rules.md` → Card gate). Old prose-only
|
|
47
|
+
evidence still passes `flow.sh check` (the floor is unchanged). **Escalate to full
|
|
48
|
+
structured lineage evidence (B1)** only if hollow-done decoys of this class recur in
|
|
49
|
+
dogfood or live eval after this addendum lands.
|
|
43
50
|
|
|
44
51
|
## Bug-fix cards: prove the test was tied to the bug
|
|
45
52
|
When a card's job is fixing a bug/regression (not new behavior), a passing test is not yet
|
|
@@ -69,6 +69,8 @@ HARNESS_PY="$SCRIPT_DIR/../harness/flow_harness.py"
|
|
|
69
69
|
# so an overridden manifest can only ever point at real shipped fixtures under the real EVAL_DIR.
|
|
70
70
|
EVAL_DIR="$SCRIPT_DIR/../eval"
|
|
71
71
|
EVAL_MANIFEST="${FLOW_EVAL_MANIFEST:-$EVAL_DIR/manifest.tsv}"
|
|
72
|
+
# FLOW_EVAL_REPLAY_DIR is a TEST-ONLY override (synthetic harness-built transcripts).
|
|
73
|
+
EVAL_REPLAY_DIR="${FLOW_EVAL_REPLAY_DIR:-$EVAL_DIR/replay}"
|
|
72
74
|
GATE_RULES_FILE="$SCRIPT_DIR/../references/gate-rules.md"
|
|
73
75
|
|
|
74
76
|
# Routing eval (v0.22 concierge routing judge - a SEPARATE modality from the artifact-vs-
|
|
@@ -3302,6 +3304,47 @@ _eval_cli_version() {
|
|
|
3302
3304
|
[ -n "$v" ] && printf '%s' "$v" || printf 'unknown'
|
|
3303
3305
|
}
|
|
3304
3306
|
|
|
3307
|
+
# Replay/record helpers. Do not live inside _eval_engine_run. Meta is key=value, one per line.
|
|
3308
|
+
_eval_replay_meta_get() { # $1=key -> value (empty + return 1 if missing)
|
|
3309
|
+
local key="$1" line
|
|
3310
|
+
[ -f "$EVAL_REPLAY_DIR/meta" ] || return 1
|
|
3311
|
+
while IFS= read -r line || [ -n "$line" ]; do
|
|
3312
|
+
line="${line%$'\r'}"
|
|
3313
|
+
case "$line" in
|
|
3314
|
+
"${key}="*) printf '%s' "${line#${key}=}"; return 0 ;;
|
|
3315
|
+
esac
|
|
3316
|
+
done < "$EVAL_REPLAY_DIR/meta"
|
|
3317
|
+
return 1
|
|
3318
|
+
}
|
|
3319
|
+
|
|
3320
|
+
_eval_replay_write_meta() { # $1=nonce $2=gate_rules_sha $3=model $4=n
|
|
3321
|
+
mkdir -p "$EVAL_REPLAY_DIR" || return 1
|
|
3322
|
+
printf 'nonce=%s\ngate_rules_sha=%s\nmodel=%s\nn=%s\n' "$1" "$2" "$3" "$4" > "$EVAL_REPLAY_DIR/meta"
|
|
3323
|
+
}
|
|
3324
|
+
|
|
3325
|
+
# Persist one stripped verdict line only — never a JSON envelope (session_id/cwd stay out).
|
|
3326
|
+
_eval_fid_safe() { # $1=fid -> filename-safe token (no path separators)
|
|
3327
|
+
local s
|
|
3328
|
+
s="$(printf '%s' "$1" | tr -c 'A-Za-z0-9' '-' | sed -E 's/-+/-/g; s/^-//; s/-$//')"
|
|
3329
|
+
[ -n "$s" ] || s="_"
|
|
3330
|
+
printf '%s' "$s"
|
|
3331
|
+
}
|
|
3332
|
+
|
|
3333
|
+
_eval_record_vote() { # $1=fid $2=vote-i $3=nonce $4=FLAG|PASS|INVALID
|
|
3334
|
+
local dest="$EVAL_REPLAY_DIR/$(_eval_fid_safe "$1")"
|
|
3335
|
+
mkdir -p "$dest" || return 1
|
|
3336
|
+
printf 'GATE-EVAL-%s: %s\n' "$3" "$4" > "$dest/$2.txt"
|
|
3337
|
+
}
|
|
3338
|
+
|
|
3339
|
+
_eval_replay_vote() { # $1=fid $2=vote-i -> transcript on stdout; return 1 if missing
|
|
3340
|
+
local f="$EVAL_REPLAY_DIR/$(_eval_fid_safe "$1")/$2.txt"
|
|
3341
|
+
if [ ! -f "$f" ]; then
|
|
3342
|
+
echo "eval --replay: missing fixture $1 vote $2 at $f" >&2
|
|
3343
|
+
return 1
|
|
3344
|
+
fi
|
|
3345
|
+
cat "$f"
|
|
3346
|
+
}
|
|
3347
|
+
|
|
3305
3348
|
# Batch header/trailer: the completeness signal `--report`/drift filter on is "start present +
|
|
3306
3349
|
# done present + n_written >= n_expected". Two paths lead to a MISSING trailer (correctly
|
|
3307
3350
|
# incomplete): (a) an interrupted batch (INT/TERM trap exits before this line, per the eval
|
|
@@ -4138,6 +4181,30 @@ function cmd_eval_converge { # same arg shape as cmd_eval, called only when --st
|
|
|
4138
4181
|
return 1
|
|
4139
4182
|
}
|
|
4140
4183
|
|
|
4184
|
+
# Live-eval entry guard (macOS DEBT). Refuse artifact eval on darwin when no real
|
|
4185
|
+
# timeout/gtimeout can bound a stuck claude call. Replay (Phase 7 sets replay_mode=1)
|
|
4186
|
+
# never hits this. --report returns before the call site. Do not move this into
|
|
4187
|
+
# _eval_engine_run or _run_with_timeout.
|
|
4188
|
+
_eval_guard_unbounded_darwin() { # $1=replay_mode (0/1); return 1 = refused
|
|
4189
|
+
local replay_mode="${1:-0}"
|
|
4190
|
+
[ "$replay_mode" -eq 0 ] || return 0
|
|
4191
|
+
[ "${FLOW_EVAL_UNBOUNDED:-}" = "1" ] && return 0
|
|
4192
|
+
local darwin=0
|
|
4193
|
+
if [ "${FLOW_EVAL_FORCE_DARWIN:-}" = "1" ]; then
|
|
4194
|
+
darwin=1
|
|
4195
|
+
else
|
|
4196
|
+
case "$(uname -s 2>/dev/null)" in Darwin) darwin=1 ;; esac
|
|
4197
|
+
fi
|
|
4198
|
+
[ "$darwin" -eq 1 ] || return 0
|
|
4199
|
+
if command -v timeout >/dev/null 2>&1 || command -v gtimeout >/dev/null 2>&1; then
|
|
4200
|
+
return 0
|
|
4201
|
+
fi
|
|
4202
|
+
echo "REFUSED: live eval on macOS without timeout/gtimeout cannot bound a stuck claude call."
|
|
4203
|
+
echo " Unbounded-billing risk: a hung judge can run past --timeout (DEBT.md)."
|
|
4204
|
+
echo " Install GNU coreutils (gtimeout) or export FLOW_EVAL_UNBOUNDED=1 to accept that risk."
|
|
4205
|
+
return 1
|
|
4206
|
+
}
|
|
4207
|
+
|
|
4141
4208
|
# NOTE: defined with the `function name { }` form (no parens) rather than this file's usual
|
|
4142
4209
|
# `name() { }` style, solely so the literal 4-byte substring the shipped verb name is built
|
|
4143
4210
|
# from never appears immediately followed by '(' in the source - a blind text-pattern security
|
|
@@ -4145,7 +4212,7 @@ function cmd_eval_converge { # same arg shape as cmd_eval, called only when --st
|
|
|
4145
4212
|
# -positives on that exact byte sequence regardless of language. Purely a spelling dodge for a
|
|
4146
4213
|
# generic scanner; behavior is 100% identical to every other function in this file.
|
|
4147
4214
|
function cmd_eval {
|
|
4148
|
-
local stage_filter="" fixture_filter="" n=3 timeout=120 report_mode=0 keep_going=0
|
|
4215
|
+
local stage_filter="" fixture_filter="" n=3 timeout=120 report_mode=0 keep_going=0 replay_mode=0 record_mode=0
|
|
4149
4216
|
while [ $# -gt 0 ]; do
|
|
4150
4217
|
case "$1" in
|
|
4151
4218
|
--stage) shift; stage_filter="${1:-}" ;;
|
|
@@ -4153,8 +4220,10 @@ function cmd_eval {
|
|
|
4153
4220
|
--n) shift; n="${1:-3}" ;;
|
|
4154
4221
|
--timeout) shift; timeout="${1:-120}" ;;
|
|
4155
4222
|
--report) report_mode=1 ;;
|
|
4223
|
+
--record) record_mode=1 ;;
|
|
4224
|
+
--replay) replay_mode=1 ;;
|
|
4156
4225
|
--keep-going) keep_going=1 ;;
|
|
4157
|
-
*) echo "usage: /flow eval [--stage 01|02|05|card|routing|converge] [--fixture <id>] [--n 3] [--timeout <seconds>] [--keep-going] [--report]"; return 1 ;;
|
|
4226
|
+
*) echo "usage: /flow eval [--stage 01|02|05|card|routing|converge] [--fixture <id>] [--n 3] [--timeout <seconds>] [--keep-going] [--report] [--record|--replay]"; return 1 ;;
|
|
4158
4227
|
esac
|
|
4159
4228
|
shift 2>/dev/null || true
|
|
4160
4229
|
done
|
|
@@ -4167,6 +4236,26 @@ function cmd_eval {
|
|
|
4167
4236
|
local backoff="${FLOW_EVAL_RETRY_BACKOFF:-5}"
|
|
4168
4237
|
case "$backoff" in ''|*[!0-9]*) backoff=5 ;; esac
|
|
4169
4238
|
|
|
4239
|
+
# Reject window: --record|--replay is artifact-modality only. Must run BEFORE routing/
|
|
4240
|
+
# converge dispatch and BEFORE the --report early-return. --record is LIVE (replay_mode
|
|
4241
|
+
# stays 0) so the Phase 6 darwin guard still fires.
|
|
4242
|
+
if [ "$record_mode" -eq 1 ] && [ "$replay_mode" -eq 1 ]; then
|
|
4243
|
+
echo "usage: /flow eval --record and --replay are mutually exclusive"
|
|
4244
|
+
return 1
|
|
4245
|
+
fi
|
|
4246
|
+
if [ "$record_mode" -eq 1 ] || [ "$replay_mode" -eq 1 ]; then
|
|
4247
|
+
case "$stage_filter" in
|
|
4248
|
+
routing|converge)
|
|
4249
|
+
echo "usage: /flow eval --record|--replay is artifact-modality only (not --stage $stage_filter)"
|
|
4250
|
+
return 1
|
|
4251
|
+
;;
|
|
4252
|
+
esac
|
|
4253
|
+
if [ "$report_mode" -eq 1 ]; then
|
|
4254
|
+
echo "usage: /flow eval --record|--replay cannot combine with --report"
|
|
4255
|
+
return 1
|
|
4256
|
+
fi
|
|
4257
|
+
fi
|
|
4258
|
+
|
|
4170
4259
|
# v0.22: routing is a SEPARATE judge modality (own manifest/prompt/verdict/results-file/
|
|
4171
4260
|
# scorecard) - delegate entirely and return, before any of the artifact-eval logic below
|
|
4172
4261
|
# (which assumes gate-rules.md stage sections and FLAG/PASS verdicts) ever runs.
|
|
@@ -4200,22 +4289,52 @@ function cmd_eval {
|
|
|
4200
4289
|
return 1
|
|
4201
4290
|
fi
|
|
4202
4291
|
|
|
4203
|
-
|
|
4204
|
-
|
|
4205
|
-
|
|
4206
|
-
|
|
4207
|
-
|
|
4208
|
-
fail)
|
|
4209
|
-
echo "SKIP: 'claude' CLI is present but the sentinel probe did not come back clean (one minimal"
|
|
4210
|
-
echo " billable probe call was made). Confirm 'claude -p' runs headless on this machine, then retry."
|
|
4211
|
-
return 0 ;;
|
|
4212
|
-
esac
|
|
4292
|
+
# Live-eval entry. replay_mode=1 self-skips inside the guard; --record leaves
|
|
4293
|
+
# replay_mode=0 so refuse-by-default still fires. Do not wrap this call again.
|
|
4294
|
+
if ! _eval_guard_unbounded_darwin "$replay_mode"; then
|
|
4295
|
+
return 1
|
|
4296
|
+
fi
|
|
4213
4297
|
|
|
4214
|
-
local nonce
|
|
4215
|
-
local total_mismatch=0 total_unreliable=0 total_evaluated=0 rundir="" n_written=0 aborted=0
|
|
4298
|
+
local nonce="" total_mismatch=0 total_unreliable=0 total_evaluated=0 rundir="" n_written=0 aborted=0
|
|
4216
4299
|
# Computed ONCE per batch, not per fixture row - see _eval_emit_result's comment.
|
|
4217
|
-
local batch_cliv batch_flowv batch_grsha
|
|
4218
|
-
|
|
4300
|
+
local batch_cliv="" batch_flowv="" batch_grsha=""
|
|
4301
|
+
if [ "$replay_mode" -eq 1 ]; then
|
|
4302
|
+
# Replay wrap: skip probe SKIP-exit and _eval_cli_version only. Still compute
|
|
4303
|
+
# batch_grsha (staleness hard-fail) and _flow_version. Nonce from fixture, never _eval_nonce.
|
|
4304
|
+
if [ ! -f "$EVAL_REPLAY_DIR/meta" ]; then
|
|
4305
|
+
echo "eval --replay: missing replay fixtures at $EVAL_REPLAY_DIR (no meta). Never SKIP."
|
|
4306
|
+
return 1
|
|
4307
|
+
fi
|
|
4308
|
+
nonce="$(_eval_replay_meta_get nonce)" || nonce=""
|
|
4309
|
+
if [ -z "$nonce" ]; then
|
|
4310
|
+
echo "eval --replay: meta missing nonce. Never SKIP."
|
|
4311
|
+
return 1
|
|
4312
|
+
fi
|
|
4313
|
+
batch_grsha="$(_eval_gate_rules_sha)"
|
|
4314
|
+
batch_flowv="$(_flow_version)"
|
|
4315
|
+
batch_cliv="replay"
|
|
4316
|
+
local rec_sha
|
|
4317
|
+
rec_sha="$(_eval_replay_meta_get gate_rules_sha)" || rec_sha=""
|
|
4318
|
+
if [ "$rec_sha" != "$batch_grsha" ]; then
|
|
4319
|
+
echo "eval --replay: fixtures stale — gate-rules changed; re-record live per ADR re-baseline rule"
|
|
4320
|
+
echo " recorded=$rec_sha current=$batch_grsha"
|
|
4321
|
+
return 1
|
|
4322
|
+
fi
|
|
4323
|
+
echo "replay: keyless artifact batch (nonce from fixture, zero live calls)"
|
|
4324
|
+
else
|
|
4325
|
+
local probe; probe="$(_eval_probe)"
|
|
4326
|
+
case "$probe" in
|
|
4327
|
+
absent)
|
|
4328
|
+
echo "SKIP: 'claude' CLI not found on PATH - eval needs it to run the LLM judge. Zero calls made."
|
|
4329
|
+
return 0 ;;
|
|
4330
|
+
fail)
|
|
4331
|
+
echo "SKIP: 'claude' CLI is present but the sentinel probe did not come back clean (one minimal"
|
|
4332
|
+
echo " billable probe call was made). Confirm 'claude -p' runs headless on this machine, then retry."
|
|
4333
|
+
return 0 ;;
|
|
4334
|
+
esac
|
|
4335
|
+
nonce="$(_eval_nonce)"
|
|
4336
|
+
batch_cliv="$(_eval_cli_version)"; batch_flowv="$(_flow_version)"; batch_grsha="$(_eval_gate_rules_sha)"
|
|
4337
|
+
fi
|
|
4219
4338
|
echo "eval: batch $nonce (N=$n per fixture, timeout=${timeout}s)"
|
|
4220
4339
|
|
|
4221
4340
|
# Ensure .flow/ (which now hosts eval-raw/ postmortem dumps) is git-ignored on any project
|
|
@@ -4246,7 +4365,9 @@ function cmd_eval {
|
|
|
4246
4365
|
}
|
|
4247
4366
|
END { print c + 0 }
|
|
4248
4367
|
' "$EVAL_MANIFEST")"
|
|
4249
|
-
|
|
4368
|
+
if [ "$replay_mode" -eq 0 ]; then
|
|
4369
|
+
_eval_emit_batch_marker "$nonce" start "$n_expected"
|
|
4370
|
+
fi
|
|
4250
4371
|
|
|
4251
4372
|
# Scoped INT/TERM trap: the sole EXIT trap (_log_on_exit -> _cleanup_tds) was found NOT to
|
|
4252
4373
|
# fire reliably while blocked on a foreground `_run_with_timeout` child (verified: a signal
|
|
@@ -4293,6 +4414,11 @@ function cmd_eval {
|
|
|
4293
4414
|
|
|
4294
4415
|
local promptfile="$rundir/prompt.txt"
|
|
4295
4416
|
if ! _eval_build_prompt "$promptfile" "$fstage_short" "$artifact_path" "$nonce"; then
|
|
4417
|
+
if [ "$replay_mode" -eq 1 ]; then
|
|
4418
|
+
echo " $fid: SKIP - no heading-mapped section (artifact-replay scope is 01|02|card)"
|
|
4419
|
+
rm -rf "$rundir" 2>/dev/null
|
|
4420
|
+
continue
|
|
4421
|
+
fi
|
|
4296
4422
|
echo " $fid: FAIL - gate-rules.md section for stage '$fstage' extracted EMPTY (heading map or file drift) - aborted before any billable call"
|
|
4297
4423
|
total_mismatch=$((total_mismatch + 1))
|
|
4298
4424
|
rm -rf "$rundir" 2>/dev/null
|
|
@@ -4312,7 +4438,12 @@ function cmd_eval {
|
|
|
4312
4438
|
# command string _eval_engine_run passes to _run_with_timeout (that broke the watchdog
|
|
4313
4439
|
# fallback on the timeout-less-PATH lane - see engine-run's note).
|
|
4314
4440
|
err1="$rundir/v${i}-a1.err"
|
|
4315
|
-
|
|
4441
|
+
if [ "$replay_mode" -eq 1 ]; then
|
|
4442
|
+
raw="$(_eval_replay_vote "$fid" "$i")" || { rm -rf "$rundir" 2>/dev/null; trap - INT TERM; return 1; }
|
|
4443
|
+
rc=0
|
|
4444
|
+
else
|
|
4445
|
+
raw="$(_eval_engine_run "$promptfile" "$timeout" 2>"$err1")"; rc=$?
|
|
4446
|
+
fi
|
|
4316
4447
|
raw1="$raw"; rc1="$rc"
|
|
4317
4448
|
if [ "$rc" -eq 124 ]; then
|
|
4318
4449
|
v=INVALID
|
|
@@ -4332,7 +4463,7 @@ function cmd_eval {
|
|
|
4332
4463
|
# _run_with_timeout is documented not to bound a stuck call (DEBT.md), a retry here just
|
|
4333
4464
|
# waits another full stuck-call duration - test E on the 3-OS CI matrix reproduces this at
|
|
4334
4465
|
# ~66s (2x ~30s stuck calls + backoff) despite --timeout 3.
|
|
4335
|
-
if [ "$v" = "INVALID" ] && [ "$rl1" != "true" ] && [ "$rc1" -ne 124 ]; then
|
|
4466
|
+
if [ "$replay_mode" -eq 0 ] && [ "$v" = "INVALID" ] && [ "$rl1" != "true" ] && [ "$rc1" -ne 124 ]; then
|
|
4336
4467
|
# Backoff then retry - env-injectable so tests can zero it; skipped when rc=124 (timeout
|
|
4337
4468
|
# was infra, retry would timeout again) or when rate-limited (see above).
|
|
4338
4469
|
if [ "$backoff" -gt 0 ]; then
|
|
@@ -4342,7 +4473,12 @@ function cmd_eval {
|
|
|
4342
4473
|
echo " $fid: retrying vote $i (parse-INVALID on attempt 1)"
|
|
4343
4474
|
fi
|
|
4344
4475
|
err2="$rundir/v${i}-a2.err"
|
|
4345
|
-
|
|
4476
|
+
if [ "$replay_mode" -eq 1 ]; then
|
|
4477
|
+
raw="$(_eval_replay_vote "$fid" "$i")" || { rm -rf "$rundir" 2>/dev/null; trap - INT TERM; return 1; }
|
|
4478
|
+
rc=0
|
|
4479
|
+
else
|
|
4480
|
+
raw="$(_eval_engine_run "$promptfile" "$timeout" 2>"$err2")"; rc=$?
|
|
4481
|
+
fi
|
|
4346
4482
|
retries_sum=$((retries_sum + 1))
|
|
4347
4483
|
if [ "$rc" -eq 124 ]; then v=INVALID; else v="$(_eval_parse_verdict "$raw" "$nonce")"; fi
|
|
4348
4484
|
[ "$model" = "unknown" ] && [ "$rc" -ne 124 ] && model="$(_eval_parse_model "$raw")"
|
|
@@ -4385,6 +4521,9 @@ function cmd_eval {
|
|
|
4385
4521
|
PASS) pass_count=$((pass_count + 1)) ;;
|
|
4386
4522
|
*) invalid_count=$((invalid_count + 1)) ;;
|
|
4387
4523
|
esac
|
|
4524
|
+
if [ "$record_mode" -eq 1 ]; then
|
|
4525
|
+
_eval_record_vote "$fid" "$i" "$nonce" "$v"
|
|
4526
|
+
fi
|
|
4388
4527
|
i=$((i + 1))
|
|
4389
4528
|
done
|
|
4390
4529
|
rm -rf "$rundir" 2>/dev/null
|
|
@@ -4415,7 +4554,9 @@ function cmd_eval {
|
|
|
4415
4554
|
echo " $fid ($fstage): $verdict - MISMATCH, expected $fexpected (flag=$flag_count pass=$pass_count invalid=$invalid_count)"
|
|
4416
4555
|
fi
|
|
4417
4556
|
|
|
4418
|
-
|
|
4557
|
+
if [ "$replay_mode" -eq 0 ]; then
|
|
4558
|
+
_eval_emit_result "$nonce" "$fid" "$fstage" "$fexpected" "$verdict" "$match" "$flag_count" "$pass_count" "$invalid_count" "$n" "$model" "$batch_cliv" "$batch_flowv" "$batch_grsha" "$retries_sum" "$vote_rate_limited"
|
|
4559
|
+
fi
|
|
4419
4560
|
n_written=$((n_written + 1))
|
|
4420
4561
|
|
|
4421
4562
|
# Circuit breaker: catches the 260710-class INVALID storm early. Trip when the FIRST
|
|
@@ -4442,9 +4583,12 @@ function cmd_eval {
|
|
|
4442
4583
|
# SKIPS the trailer so an early-broken batch never satisfies --report's completeness check
|
|
4443
4584
|
# (n_written>=n_expected AND trailer present), even on a --fixture-filtered run where
|
|
4444
4585
|
# n_written could accidentally equal the filtered n_expected of 1.
|
|
4445
|
-
if [ "$aborted" -eq 0 ]; then
|
|
4586
|
+
if [ "$aborted" -eq 0 ] && [ "$replay_mode" -eq 0 ]; then
|
|
4446
4587
|
_eval_emit_batch_marker "$nonce" done "$n_written"
|
|
4447
4588
|
fi
|
|
4589
|
+
if [ "$record_mode" -eq 1 ] && [ "$aborted" -eq 0 ]; then
|
|
4590
|
+
_eval_replay_write_meta "$nonce" "$batch_grsha" "${model:-unknown}" "$n"
|
|
4591
|
+
fi
|
|
4448
4592
|
|
|
4449
4593
|
echo
|
|
4450
4594
|
if [ "$total_evaluated" -eq 0 ]; then
|
|
@@ -4457,8 +4601,13 @@ function cmd_eval {
|
|
|
4457
4601
|
echo "ABORTED: circuit breaker tripped on first fixture UNRELIABLE - $total_evaluated of $n_expected fixtures evaluated."
|
|
4458
4602
|
return 2
|
|
4459
4603
|
fi
|
|
4460
|
-
|
|
4604
|
+
if [ "$replay_mode" -eq 0 ]; then
|
|
4605
|
+
_eval_print_scorecard "$nonce"
|
|
4606
|
+
fi
|
|
4461
4607
|
echo
|
|
4608
|
+
if [ "$replay_mode" -eq 1 ]; then
|
|
4609
|
+
echo "replay: $total_evaluated fixture(s) (keyless)"
|
|
4610
|
+
fi
|
|
4462
4611
|
if [ "$total_mismatch" -eq 0 ] && [ "$total_unreliable" -eq 0 ]; then
|
|
4463
4612
|
echo "PASS: all $total_evaluated evaluated fixture(s) majority-matched their expected verdict."
|
|
4464
4613
|
return 0
|
|
@@ -4506,9 +4655,14 @@ usage: bash flow.sh <command> [args]
|
|
|
4506
4655
|
promote <file> Copy a playbook into the cross-project KB (~/.claude/flow/playbooks)
|
|
4507
4656
|
doctor Check the environment (bash/python/grep/git) across macOS/Linux/Windows
|
|
4508
4657
|
eval [opts] Behavioral eval: does the LLM semantic gate flag a hollow-but-mechanically-clean
|
|
4509
|
-
fixture? Opt-in, BILLABLE (skips cleanly with zero calls if 'claude'
|
|
4658
|
+
fixture? Opt-in, BILLABLE (live mode skips cleanly with zero calls if 'claude'
|
|
4659
|
+
CLI absent; --replay never inherits that SKIP).
|
|
4510
4660
|
[--stage 01|02|05|card|routing|converge] [--fixture <id>] [--n 3] [--timeout <seconds>]
|
|
4511
4661
|
[--report] offline: last complete batch's scorecard + drift, zero calls
|
|
4662
|
+
[--record] live: write stripped transcripts under eval/replay/ (billable)
|
|
4663
|
+
[--replay] keyless: feed recorded transcripts through parse/vote/scorecard
|
|
4664
|
+
--record|--replay are artifact-modality only (not --stage routing|converge,
|
|
4665
|
+
not --report). --replay is not a fresh-judge and never counts toward the floor.
|
|
4512
4666
|
--stage routing: SEPARATE judge (concierge routing, not gate-rules FLAG/PASS);
|
|
4513
4667
|
hard-capped at 90 calls/batch, own results stream, own --report
|
|
4514
4668
|
retro Print the 3 retro questions
|
package/skills-manifest.json
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
{
|
|
2
2
|
"source": "../skills/flow",
|
|
3
3
|
"destRelative": "skills/flow",
|
|
4
|
-
"fileCount":
|
|
5
|
-
"syncedAt": "2026-08-
|
|
4
|
+
"fileCount": 102,
|
|
5
|
+
"syncedAt": "2026-08-14T09:19:34.115Z",
|
|
6
6
|
"fileList": [
|
|
7
7
|
"SKILL.md",
|
|
8
8
|
"_templates/00-idea.md",
|
|
@@ -35,6 +35,8 @@
|
|
|
35
35
|
"eval/fixtures/fcdb/cards/C-001.md",
|
|
36
36
|
"eval/fixtures/fcdc/.flow/.gitkeep",
|
|
37
37
|
"eval/fixtures/fcdc/cards/C-001.md",
|
|
38
|
+
"eval/fixtures/fcdd/cards/C-001.md",
|
|
39
|
+
"eval/fixtures/fcde/cards/C-001.md",
|
|
38
40
|
"eval/fixtures/routing/manifest.tsv",
|
|
39
41
|
"eval/fixtures/routing/states/state-any-midproject.txt",
|
|
40
42
|
"eval/fixtures/routing/states/state-cards-todo.txt",
|