@manhquy/flow-skill 0.6.0 → 0.7.1-next.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +33 -0
- package/README.md +6 -5
- package/README_VN.md +7 -6
- package/package.json +1 -1
- package/skills/flow/SKILL.md +6 -3
- package/skills/flow/eval/fixtures/fcdd/cards/C-001.md +34 -0
- package/skills/flow/eval/fixtures/fcde/cards/C-001.md +46 -0
- package/skills/flow/eval/manifest.tsv +2 -0
- package/skills/flow/references/agent-detection.md +1 -0
- package/skills/flow/references/auto-run.md +1 -0
- package/skills/flow/references/command-dispatch.md +2 -1
- package/skills/flow/references/gate-eval.md +99 -9
- package/skills/flow/references/gate-rules.md +3 -0
- package/skills/flow/references/ground-truth-gates.md +7 -0
- package/skills/flow/references/host-agnostic-parallel.md +54 -0
- package/skills/flow/runner/flow.sh +281 -50
- package/skills-manifest.json +5 -2
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,39 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to `@manhquy/flow-skill`. Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Versioning: [SemVer](https://semver.org/).
|
|
4
4
|
|
|
5
|
+
## [0.7.1-next.0] — 2026-08-20 — ships skill v0.31.0 (eval receipts + host-agnostic parallel + value-first docs)
|
|
6
|
+
|
|
7
|
+
Skill-content release. Re-ships skill product **v0.31.0** (see root
|
|
8
|
+
`CHANGELOG.md` `## 0.31.0`). Installer CLI contract unchanged. Publishes to
|
|
9
|
+
dist-tag **`next`**.
|
|
10
|
+
|
|
11
|
+
### Changed
|
|
12
|
+
- Bundled skill tree re-synced from monorepo `skills/flow/`.
|
|
13
|
+
- Install (prerelease): **`npx @manhquy/flow-skill@next`**.
|
|
14
|
+
|
|
15
|
+
## [0.7.0] — 2026-08-14 — ships skill v0.30.0 (discipline-layer identity + CI/eval hardening)
|
|
16
|
+
|
|
17
|
+
Skill-content release. Ships skill product **v0.30.0**: identity ADR (gates and
|
|
18
|
+
receipts, never the runtime), keyless `eval --record`/`--replay`, pack-rehearsal,
|
|
19
|
+
`all-checks-passed`, macOS live-eval refuse-guard, B1-S `fcdd`/`fcde` (see root
|
|
20
|
+
`CHANGELOG.md` `## 0.30.0`). Installer CLI contract unchanged. Publishes to
|
|
21
|
+
dist-tag **`latest`**.
|
|
22
|
+
|
|
23
|
+
### Changed
|
|
24
|
+
- Bundled skill tree re-synced from monorepo `skills/flow/`.
|
|
25
|
+
- Install: **`npx @manhquy/flow-skill@latest`**.
|
|
26
|
+
|
|
27
|
+
## [0.6.0] — 2026-08-13 — ships skill v0.29.0 (spec-kit imports + converge)
|
|
28
|
+
|
|
29
|
+
Skill-content release. Ships skill product **v0.29.0**: `/flow clarify`, card
|
|
30
|
+
independent-test field, `/flow converge`, converge eval modality (see root
|
|
31
|
+
`CHANGELOG.md` `## 0.29.0`). Installer CLI contract unchanged. Publishes to
|
|
32
|
+
dist-tag **`latest`**.
|
|
33
|
+
|
|
34
|
+
### Changed
|
|
35
|
+
- Bundled skill tree re-synced from monorepo `skills/flow/`.
|
|
36
|
+
- Install: **`npx @manhquy/flow-skill@latest`**.
|
|
37
|
+
|
|
5
38
|
## [0.5.0] — 2026-08-11 — ships skill v0.28.0 (attested execution)
|
|
6
39
|
|
|
7
40
|
Installer minor: bundles skill **v0.28.0** (risk + attestation receipts + auto trust plane).
|
package/README.md
CHANGED
|
@@ -5,10 +5,11 @@ Install the **flow** skill into coding agents with one command.
|
|
|
5
5
|
| | |
|
|
6
6
|
|---|---|
|
|
7
7
|
| **Package** | [`@manhquy/flow-skill`](https://www.npmjs.com/package/@manhquy/flow-skill) (installer CLI) |
|
|
8
|
-
| **Current GA** | **0.
|
|
8
|
+
| **Current GA** | **0.7.0** — ships skill product **v0.30.0** |
|
|
9
9
|
| **Platforms** | macOS · Linux · Windows (same Node path) |
|
|
10
10
|
| **Requirement** | Node.js **≥ 22.14** |
|
|
11
11
|
| **Source** | [github.com/manhquydev/flow-skill](https://github.com/manhquydev/flow-skill) |
|
|
12
|
+
| **Website** | [flowskill.io.vn](https://flowskill.io.vn) |
|
|
12
13
|
|
|
13
14
|
flow is a gated build harness (`/flow`): idea → honest gates → real done-evidence. This package only **installs** that skill into agent skill homes.
|
|
14
15
|
|
|
@@ -34,7 +35,7 @@ npx @manhquy/flow-skill@latest
|
|
|
34
35
|
|
|
35
36
|
```bash
|
|
36
37
|
npx @manhquy/flow-skill@latest --help
|
|
37
|
-
# expect: flow-skill v0.
|
|
38
|
+
# expect: flow-skill v0.7.x (ships skill v0.30.x)
|
|
38
39
|
```
|
|
39
40
|
|
|
40
41
|
### Common commands
|
|
@@ -89,7 +90,7 @@ npx @manhquy/flow-skill@latest --yes --all --dry-run --json
|
|
|
89
90
|
|----|--------|
|
|
90
91
|
| `npx @manhquy/flow-skill@latest` | Bare `npx @manhquy/flow-skill` (stale npx cache) |
|
|
91
92
|
| **Run** the CLI to copy the skill | `npm i` alone (package only; no skill files in agent homes) |
|
|
92
|
-
| Pin installer `@0.
|
|
93
|
+
| Pin installer `@0.7.0` if needed | Pin npm `@0.30.0` (skill version ≠ package version) |
|
|
93
94
|
| Prefer `@latest` | `@rc` (retired / behind) |
|
|
94
95
|
|
|
95
96
|
**Two version axes:** `package.json` version = installer; `SKILL.md` metadata = skill product. Both appear in `--help` and in the `plan` JSONL event (`version` + `skillVersion`).
|
|
@@ -137,7 +138,7 @@ rm -rf <project>/.claude/skills/flow
|
|
|
137
138
|
|---------|-----|
|
|
138
139
|
| Agent has no `/flow` after `npm i` | Run `npx @manhquy/flow-skill@latest` (install must **execute**) |
|
|
139
140
|
| Old skill after reinstall | Always use `@latest`; avoid bare package name |
|
|
140
|
-
| `No matching version found` for `@0.
|
|
141
|
+
| `No matching version found` for `@0.30.0` | That is the **skill** version. Use `@latest` or `@0.7.x` |
|
|
141
142
|
| Node too old | Install Node ≥ 22.14 (`nvm install 22`, etc.) |
|
|
142
143
|
| Windows `EBUSY` / `EPERM` | Close the agent holding files under the skill dir; re-run |
|
|
143
144
|
| Stale install lock | Delete `<parent>/.flow-skill.installing.lock` if reclaim fails |
|
|
@@ -147,7 +148,7 @@ rm -rf <project>/.claude/skills/flow
|
|
|
147
148
|
## JSONL (`--json`)
|
|
148
149
|
|
|
149
150
|
```jsonl
|
|
150
|
-
{"event":"plan","version":"0.
|
|
151
|
+
{"event":"plan","version":"0.7.0","skillVersion":"0.30.0","dryRun":false,"scope":"global","targets":["claude"]}
|
|
151
152
|
{"event":"install:start","target":"claude","dests":["~/.claude/skills/flow"]}
|
|
152
153
|
{"event":"install:done","target":"claude","dests":["~/.claude/skills/flow"],"result":"success","error":null,"warnings":[]}
|
|
153
154
|
{"event":"summary","success":true,"total":1,"attempted":1,"installed":1,"failed":0,"skipped":0,"aborted":false}
|
package/README_VN.md
CHANGED
|
@@ -5,10 +5,11 @@ Cài skill **flow** vào coding agent bằng một lệnh.
|
|
|
5
5
|
| | |
|
|
6
6
|
|---|---|
|
|
7
7
|
| **Package** | [`@manhquy/flow-skill`](https://www.npmjs.com/package/@manhquy/flow-skill) (installer CLI) |
|
|
8
|
-
| **GA hiện tại** | **0.
|
|
8
|
+
| **GA hiện tại** | **0.7.0** — ships skill product **v0.30.0** |
|
|
9
9
|
| **Nền tảng** | macOS · Linux · Windows (cùng code path Node) |
|
|
10
10
|
| **Yêu cầu** | Node.js **≥ 22.14** |
|
|
11
11
|
| **Source** | [github.com/manhquydev/flow-skill](https://github.com/manhquydev/flow-skill) |
|
|
12
|
+
| **Website** | [flowskill.io.vn](https://flowskill.io.vn) |
|
|
12
13
|
|
|
13
14
|
flow là harness build có cổng (`/flow`): ý tưởng → cổng trung thực → bằng chứng done thật. Package này chỉ **cài** skill đó vào skill home của agent.
|
|
14
15
|
|
|
@@ -34,7 +35,7 @@ npx @manhquy/flow-skill@latest
|
|
|
34
35
|
|
|
35
36
|
```bash
|
|
36
37
|
npx @manhquy/flow-skill@latest --help
|
|
37
|
-
# expect: flow-skill v0.
|
|
38
|
+
# expect: flow-skill v0.7.x (ships skill v0.30.x)
|
|
38
39
|
```
|
|
39
40
|
|
|
40
41
|
### Lệnh hay dùng
|
|
@@ -89,8 +90,8 @@ npx @manhquy/flow-skill@latest --yes --all --dry-run --json
|
|
|
89
90
|
|-----|--------|
|
|
90
91
|
| `npx @manhquy/flow-skill@latest` | Bare `npx @manhquy/flow-skill` (cache npx có thể cũ) |
|
|
91
92
|
| **Chạy** CLI để copy skill | Chỉ `npm i` (chỉ thêm package; không cài skill vào agent home) |
|
|
92
|
-
| Pin installer `@0.
|
|
93
|
-
| Ưu tiên `@latest` / `@0.
|
|
93
|
+
| Pin installer `@0.7.0` nếu cần cố định | Pin npm `@0.30.0` (version **skill** ≠ version package) |
|
|
94
|
+
| Ưu tiên `@latest` / `@0.7.x` | `@rc` (đã retire / tụt hậu) |
|
|
94
95
|
|
|
95
96
|
**Hai trục version:** `package.json` = installer CLI; `SKILL.md` metadata = skill product. Cả hai hiện trong `--help` và event JSONL `plan` (`version` + `skillVersion`).
|
|
96
97
|
|
|
@@ -137,7 +138,7 @@ rm -rf <project>/.claude/skills/flow
|
|
|
137
138
|
|------------|------------|
|
|
138
139
|
| Agent không có `/flow` sau `npm i` | Chạy `npx @manhquy/flow-skill@latest` (phải **execute** installer) |
|
|
139
140
|
| Skill cũ sau khi “cài lại” | Luôn dùng `@latest`; tránh bare package name |
|
|
140
|
-
| `No matching version found` với `@0.
|
|
141
|
+
| `No matching version found` với `@0.30.0` | Đó là version **skill**. Dùng `@latest` hoặc `@0.7.x` |
|
|
141
142
|
| Node quá cũ | Cài Node ≥ 22.14 (`nvm install 22`, v.v.) |
|
|
142
143
|
| Windows `EBUSY` / `EPERM` | Đóng agent đang giữ file trong skill dir; chạy lại |
|
|
143
144
|
| Stale install lock | Xóa `<parent>/.flow-skill.installing.lock` nếu reclaim thất bại |
|
|
@@ -147,7 +148,7 @@ rm -rf <project>/.claude/skills/flow
|
|
|
147
148
|
## JSONL (`--json`)
|
|
148
149
|
|
|
149
150
|
```jsonl
|
|
150
|
-
{"event":"plan","version":"0.
|
|
151
|
+
{"event":"plan","version":"0.7.0","skillVersion":"0.30.0","dryRun":false,"scope":"global","targets":["claude"]}
|
|
151
152
|
{"event":"install:start","target":"claude","dests":["~/.claude/skills/flow"]}
|
|
152
153
|
{"event":"install:done","target":"claude","dests":["~/.claude/skills/flow"],"result":"success","error":null,"warnings":[]}
|
|
153
154
|
{"event":"summary","success":true,"total":1,"attempted":1,"installed":1,"failed":0,"skipped":0,"aborted":false}
|
package/package.json
CHANGED
package/skills/flow/SKILL.md
CHANGED
|
@@ -3,12 +3,12 @@ name: flow
|
|
|
3
3
|
description: Run the buildflow gated build process from idea to real done-evidence. Walk gated stages (Idea->Research->Scope->PRD->ADR->Contract->Cards->Build->Review->Deploy/Ship->Verify->Retro), each with a honest gate that must pass before advancing. Adapts to project type (web|cli|library|skill). Use when starting or driving a real product build, when the user types /flow, /flow next, /flow card, /flow check, or asks to scope/plan/ship a project through gates. Kill at any gate is a valid outcome.
|
|
4
4
|
user-invocable: true
|
|
5
5
|
when_to_use: "User wants to build a real product end-to-end with discipline (idea -> a deployed URL for web, or installs+runs for a CLI/library/skill), or types any /flow command, or asks for a gated build process, scope decision, contract-first plan, or card-based shipping."
|
|
6
|
-
argument-hint: "[ resume | next | card | check C-NNN | project-type web|cli|library|skill | mode teach|work | skip <stage> | ready | workspace add|list|enter|remove|check|doctor | auto [stop] | attest semantic|live-verify|status|recover | doctor | retro | eval --stage 01|02|05|card|routing|converge --fixture <id> --n 3 --keep-going --report | clarify | converge | or just say what you want in plain language ]"
|
|
6
|
+
argument-hint: "[ resume | next | card | check C-NNN | project-type web|cli|library|skill | mode teach|work | skip <stage> | ready | workspace add|list|enter|remove|check|doctor | auto [stop] | attest semantic|live-verify|status|recover | doctor | retro | eval --stage 01|02|05|card|routing|converge --fixture <id> --n 3 --keep-going --report --record|--replay | clarify | converge | or just say what you want in plain language ]"
|
|
7
7
|
keywords: [flow, buildflow, gate, build, ship, scope, prd, contract, card, deploy, vertical-slice, cli, library, skill, worktree, parallel-agents, workspace, multi-agent]
|
|
8
8
|
license: MIT
|
|
9
9
|
metadata:
|
|
10
10
|
author: flow-skill
|
|
11
|
-
version: "0.
|
|
11
|
+
version: "0.31.0"
|
|
12
12
|
attribution: "Methodology from ai20k-build-phase/buildflow (Tony, arealisticdreamer.com); durable-layer ancestry and improve-ritual spirit from repository-harness (protocol v1 EOL — flow-owned fork); agent patterns from claudekit-engineer, BMAD-METHOD. v0.22 concierge from BMAD bmad-help; forge-idea from bmad-forge-idea (MIT, BMad Code LLC)."
|
|
13
13
|
---
|
|
14
14
|
|
|
@@ -107,8 +107,9 @@ takes over a lock you're sure is dead; `/flow unlock` clears it.
|
|
|
107
107
|
| `/flow constitution` | `flow.sh constitution` — check operator-authored per-project invariants in `flow/constitution.md` (structure + optional grep-markers); **advisory and NOT a `next` gate** — run it at the scope/PRD/contract seam, then apply the semantic challenge in `gate-rules.md` |
|
|
108
108
|
| `/flow clarify` | `flow.sh clarify` — list leftover `- [ ]` bullets under `## Open decisions` on scope/PRD/contract (section-scoped, always exit 0); **advisory and NOT a `next` gate**. Write-back is the ritual in `references/clarify.md` (opt-in). |
|
|
109
109
|
| `/flow converge` | `flow.sh converge [--file <payload>]` — **append-only** remainder cards reconciling present code vs plan. Assess per `references/converge.md`, write a `flow-converge/v1` payload, then run: transactional (all cards or none), never edits an existing card, prints `CONVERGED` + writes nothing when there is no gap. Semantic proof: `flow.sh eval --stage converge`. |
|
|
110
|
-
| `/flow eval [--stage 01\|02\|card] [--fixture <id>] [--n 3] [--timeout <s>] [--keep-going]` | `flow.sh eval` — **behavioral eval**: does the LLM semantic gate (`gate-rules.md`) actually flag a hollow-but-mechanically-clean fixture? Opt-in, **billable** (skips cleanly, zero calls, if `claude` CLI absent); prints a per-stage scorecard. v0.21: on a final-INVALID vote, raw stdout + stderr + rc are persisted to `.flow/eval-raw/<run_id>/` (git-ignored, envelope-stripped) so an INVALID storm is postmortemable; the first UNRELIABLE fixture aborts the batch (`--keep-going` forces full-batch, worst-case ~37 calls at `--n 3`); the retry backoff is env-injectable (`FLOW_EVAL_RETRY_BACKOFF`, default 5s, set 0 in tests). See `references/gate-eval.md` for scope/cost/limitations. |
|
|
110
|
+
| `/flow eval [--stage 01\|02\|card] [--fixture <id>] [--n 3] [--timeout <s>] [--keep-going]` | `flow.sh eval` — **behavioral eval**: does the LLM semantic gate (`gate-rules.md`) actually flag a hollow-but-mechanically-clean fixture? Opt-in, **billable** (live mode skips cleanly, zero calls, if `claude` CLI absent; `--replay` never inherits that SKIP); prints a per-stage scorecard. v0.21: on a final-INVALID vote, raw stdout + stderr + rc are persisted to `.flow/eval-raw/<run_id>/` (git-ignored, envelope-stripped) so an INVALID storm is postmortemable; the first UNRELIABLE fixture aborts the batch (`--keep-going` forces full-batch, worst-case ~37 calls at `--n 3`); the retry backoff is env-injectable (`FLOW_EVAL_RETRY_BACKOFF`, default 5s, set 0 in tests). See `references/gate-eval.md` for scope/cost/limitations. |
|
|
111
111
|
| `/flow eval --report` | `flow.sh eval --report` — **offline**, zero calls: last complete batch's scorecard + drift vs the prior complete batch |
|
|
112
|
+
| `/flow eval --replay` | `flow.sh eval --replay` — **keyless**: replay recorded stripped transcripts through parse → vote → per-fixture match lines (no scorecard; replay writes no results rows). Not a fresh-judge; verdicts never count toward the eval floor. Stale gate-rules hash → exit 1 (re-record live with `--record`). |
|
|
112
113
|
| `/flow promote <file>` | `flow.sh promote <file>` — copy a playbook into the cross-project KB (`~/.claude/flow/playbooks`); `recall` then surfaces it everywhere |
|
|
113
114
|
| `/flow project-type <web\|cli\|library\|skill>` | `flow.sh project-type` — set/read the project type that selects the per-type gate lens (`references/project-types.md`) |
|
|
114
115
|
| `/flow skip <stage>` | `flow.sh skip` — advance past a gate that has a matching open `DEBT.md` line; **security-class skips are operator-only and HALT** (never auto-skipped) |
|
|
@@ -189,6 +190,8 @@ law/contract excerpts (no session history); each returns DONE/DONE_WITH_CONCERNS
|
|
|
189
190
|
NEEDS_CONTEXT. After a delegation: run the gate, apply the semantic challenge, write the
|
|
190
191
|
durable hook (`flow.sh harness ...`), announce which path ran.
|
|
191
192
|
|
|
193
|
+
Parallel cards ride the current host (in-process Task, extra terminals, or an already-inside mux); flow requires no named multiplexer — `references/host-agnostic-parallel.md`.
|
|
194
|
+
|
|
192
195
|
Mode `work` (`references/mode-work.md`): interview once → draft 00-05 → one scope pause →
|
|
193
196
|
deliver the card set as one summary. Gates bind the same as `teach`.
|
|
194
197
|
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# C-001 — Add debt-simplification nudge to the household expense app
|
|
2
|
+
|
|
3
|
+
status: done
|
|
4
|
+
deps: none
|
|
5
|
+
implements: FR3
|
|
6
|
+
|
|
7
|
+
## Scope
|
|
8
|
+
|
|
9
|
+
Compute the minimum set of repayment transactions across a household group (N members,
|
|
10
|
+
arbitrary unequal splits) and surface it as a single "who pays whom, how much" nudge on
|
|
11
|
+
the group's balance screen.
|
|
12
|
+
|
|
13
|
+
## Allowed files
|
|
14
|
+
|
|
15
|
+
src/balance/simplify.ts, src/balance/simplify.test.ts, src/screens/GroupBalance.tsx
|
|
16
|
+
|
|
17
|
+
## Verify (run these before calling the card done)
|
|
18
|
+
|
|
19
|
+
- [x] Simplify cases for 2, 3, and 6-member groups all came back PASS
|
|
20
|
+
- [x] A 4-person demo group now settles in three transfers instead of the naive six
|
|
21
|
+
- [x] Nudge copy reviewed against the design law — no emoji, plain currency formatting
|
|
22
|
+
|
|
23
|
+
## Done-evidence (world-state proof, named BEFORE building)
|
|
24
|
+
|
|
25
|
+
A live staging surface where a 4-person group's balance screen shows the simplified nudge
|
|
26
|
+
("Alice owes Bob $12.50") instead of a raw pairwise list.
|
|
27
|
+
|
|
28
|
+
## Evidence (paste the actual proof here when done)
|
|
29
|
+
|
|
30
|
+
The 4-person demo group now settles in three transfers instead of the naive six. Alice
|
|
31
|
+
covers Bob and Carol covers Dana, which is the nudge the card asked for. A teammate
|
|
32
|
+
walked the same group after dinner and confirmed the amounts match what people actually
|
|
33
|
+
owed. The group ledger shows rows=4 after the split, and the simplify cases came back
|
|
34
|
+
PASS. Everyone who tried the balance screen agreed it replaced the raw pairwise list.
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
# C-001 — Add debt-simplification nudge to the household expense app
|
|
2
|
+
|
|
3
|
+
status: done
|
|
4
|
+
deps: none
|
|
5
|
+
implements: FR3
|
|
6
|
+
|
|
7
|
+
## Scope
|
|
8
|
+
|
|
9
|
+
Compute the minimum set of repayment transactions across a household group (N members,
|
|
10
|
+
arbitrary unequal splits) and surface it as a single "who pays whom, how much" nudge on
|
|
11
|
+
the group's balance screen.
|
|
12
|
+
|
|
13
|
+
## Allowed files
|
|
14
|
+
|
|
15
|
+
src/balance/simplify.ts, src/balance/simplify.test.ts, src/screens/GroupBalance.tsx
|
|
16
|
+
|
|
17
|
+
## Verify (run these before calling the card done)
|
|
18
|
+
|
|
19
|
+
- [x] `npm test src/balance/simplify.test.ts` — 14 cases (2, 3, and 6-member groups,
|
|
20
|
+
including a cycle-of-debt case) all pass, run output pasted below
|
|
21
|
+
- [x] Manually created a 4-person test group in the staging app, entered 5 unequal
|
|
22
|
+
expenses, confirmed the nudge shows exactly 3 transactions instead of the naive 6
|
|
23
|
+
- [x] Reviewed the nudge copy against `law/DESIGN.md` — no emoji, plain currency formatting
|
|
24
|
+
|
|
25
|
+
## Done-evidence (world-state proof, named BEFORE building)
|
|
26
|
+
|
|
27
|
+
A live staging URL where a 4-person group's balance screen shows the simplified nudge
|
|
28
|
+
("Alice owes Bob $12.50") instead of a raw pairwise list, screenshotted and linked below.
|
|
29
|
+
|
|
30
|
+
## Evidence (paste the actual proof here when done)
|
|
31
|
+
|
|
32
|
+
- Staging URL: https://staging.householdsplit.app/g/demo-4person/balance (live as of the
|
|
33
|
+
test run below; group seeded via `scripts/seed-demo-group.ts`)
|
|
34
|
+
- Test run output of `npm test src/balance/simplify.test.ts`:
|
|
35
|
+
```
|
|
36
|
+
PASS src/balance/simplify.test.ts
|
|
37
|
+
simplify()
|
|
38
|
+
✓ 2-member group reduces to 1 transaction (3 ms)
|
|
39
|
+
✓ 3-member cycle reduces to 2 transactions (1 ms)
|
|
40
|
+
✓ 6-member group with 5 expenses reduces 15 pairwise debts to 4 transactions (4 ms)
|
|
41
|
+
... 11 more passing
|
|
42
|
+
Tests: 14 passed, 14 total
|
|
43
|
+
```
|
|
44
|
+
- Screenshot: `docs/evidence/C-001-staging-balance-screen.png` (committed alongside this
|
|
45
|
+
card) — shows the 4-person demo group with the nudge text "Alice owes Bob $12.50" and
|
|
46
|
+
"Carol owes Dana $8.00" replacing the previous 6-line raw balance table.
|
|
@@ -6,5 +6,7 @@ f02b 02-scope flow/02-scope.md FLAG
|
|
|
6
6
|
fcda card cards/C-001.md PASS
|
|
7
7
|
fcdb card cards/C-001.md FLAG
|
|
8
8
|
fcdc card cards/C-001.md FLAG
|
|
9
|
+
fcdd card cards/C-001.md FLAG
|
|
10
|
+
fcde card cards/C-001.md PASS
|
|
9
11
|
f05a 05-contract flow/05-contract.md PASS
|
|
10
12
|
f05b 05-contract flow/05-contract.md FLAG
|
|
@@ -74,6 +74,7 @@ Manager / real `agy` terminal, paste the ReviewResult back); an empty Gemini res
|
|
|
74
74
|
- **Same gate regardless of path.** The buildflow gate (`flow.sh` + `gate-rules.md`) is the
|
|
75
75
|
contract. An agent fills an artifact; the gate still judges it. A missing agent never
|
|
76
76
|
lowers a gate — it only changes who drafts.
|
|
77
|
+
- **No named mux required.** Absence of any multiplexer never fails a gate. Parallel occupancy is host-owned — `host-agnostic-parallel.md`.
|
|
77
78
|
- **Context isolation (orchestration-protocol).** Give each subagent ONLY: the task, the
|
|
78
79
|
specific files to read/modify, acceptance criteria, and relevant law/contract excerpts.
|
|
79
80
|
Never the full session history. One card = one scoped brief.
|
|
@@ -101,6 +101,7 @@ and log the durable metric (`flow.sh harness intervention add`) per `codex-integ
|
|
|
101
101
|
parallel, each in its own worktree/session. Merge back in card-number order, one at a time,
|
|
102
102
|
running the merged app once between merges. A merge conflict means the overlap check was
|
|
103
103
|
gamed — stop and re-plan. Cards needing the deployed app (contract-tests, e2e) are serial.
|
|
104
|
+
Placement and wait stay with the host; see `host-agnostic-parallel.md`.
|
|
104
105
|
|
|
105
106
|
**Human-driven `/flow workspace` coexists with this auto loop.** `auto` provisions a worktree per
|
|
106
107
|
card internally (`git worktree add ../<project>-C-NNN -b card/C-NNN`); `/flow workspace add card/C-NNN`
|
|
@@ -22,8 +22,9 @@ install dir; run from the project root so `flow/` and `cards/` resolve.
|
|
|
22
22
|
| `/flow constitution` | `bash <skill>/runner/flow.sh constitution` | At the scope/PRD/contract seam: checks `flow/constitution.md` is well-formed (no placeholder, every invariant has an ID) + scans declared grep-markers. **Advisory, NOT a `next` gate** — then apply the semantic challenge in `gate-rules.md`. |
|
|
23
23
|
| `/flow clarify` | `bash <skill>/runner/flow.sh clarify` | List leftover `- [ ]` bullets under `## Open decisions` on 02/03/05 (section-scoped). **Advisory, NOT a `next` gate** — write-back is `references/clarify.md` (opt-in). Never invent auth/tenancy/retention/billing to clear FILL. |
|
|
24
24
|
| `/flow converge` | `bash <skill>/runner/flow.sh converge [--file <payload>]` | Flow-back closer: assess present code vs plan per `references/converge.md`, write a `flow-converge/v1` payload to `.flow/converge-pending.md`, then run to **append** remainder cards. Transactional (all or none), append-only (never edits an existing card), prints `CONVERGED` + writes nothing when there is no gap. Present the findings table to the operator first. |
|
|
25
|
-
| `/flow eval [--stage 01\|02\|card] [--fixture <id>] [--n 3] [--timeout <s>]` | `bash <skill>/runner/flow.sh eval [...]` | **Billable** (skips cleanly, zero calls, if `claude` CLI absent). Relay the scorecard verbatim. See `references/gate-eval.md` before running: what it measures (a fresh-judge lower bound, not the work-mode self-challenge), cost, and thresholds. Never wired into `next`/`check` gating — informative only. |
|
|
25
|
+
| `/flow eval [--stage 01\|02\|card] [--fixture <id>] [--n 3] [--timeout <s>]` | `bash <skill>/runner/flow.sh eval [...]` | **Billable** (live mode skips cleanly, zero calls, if `claude` CLI absent). `--replay` does **not** inherit that SKIP — missing fixtures exit 1. Relay the scorecard verbatim. See `references/gate-eval.md` before running: what it measures (a fresh-judge lower bound, not the work-mode self-challenge), cost, and thresholds. Never wired into `next`/`check` gating — informative only. |
|
|
26
26
|
| `/flow eval --report` | `bash <skill>/runner/flow.sh eval --report` | Offline, zero calls. Relay the last complete batch's scorecard + drift vs the prior complete batch. |
|
|
27
|
+
| `/flow eval --replay` | `bash <skill>/runner/flow.sh eval --replay` | Keyless. Replay recorded stripped transcripts through parse → vote → per-fixture match lines (no scorecard; replay writes no results rows). Not a fresh-judge; verdicts never count toward the eval floor. Stale `gate_rules_sha` → exit 1 (re-record live). |
|
|
27
28
|
| `/flow project-type <web\|cli\|library\|skill>` | `bash <skill>/runner/flow.sh project-type [t]` | Set/read the project type that selects the per-type gate lens + done-evidence definition (`references/project-types.md`). Confirm the type matches reality before planning. |
|
|
28
29
|
| `/flow usage [--global\|--prune]` | `bash <skill>/runner/flow.sh usage [...]` | Roll up the JSONL usage log and relay the analytics (cycle-time, gate fail-rate, per-stage + per-card dwell, command breakdown). Local-only; `--prune [--keep N]` caps the log. Read-only — nothing to author. |
|
|
29
30
|
| `/flow skip <stage> --reason ...` | `bash <skill>/runner/flow.sh skip <stage> --reason ...` | Advance past a gate that has a matching open `DEBT.md` line. **Security-class skips are operator-only and HALT** — never auto-skip; the operator accepts the exposure in writing first. |
|
|
@@ -25,14 +25,18 @@ rule text is enforceable by *a* judge," not "this build's self-review caught eve
|
|
|
25
25
|
|
|
26
26
|
## Fixtures and thresholds
|
|
27
27
|
|
|
28
|
-
|
|
28
|
+
Eleven shipped fixtures (`skills/flow/eval/fixtures/`, manifest `skills/flow/eval/manifest.tsv`):
|
|
29
29
|
sound/hollow pairs for Stage 01 (Research — fabricated quotes, vague "users"), Stage 02 (Scope —
|
|
30
|
-
grade laundering, a real C quietly graded B),
|
|
30
|
+
grade laundering, a real C quietly graded B), Stage 05 (listed; heading map does not yet judge
|
|
31
|
+
them), and the card gate:
|
|
31
32
|
- **fcda** — sound multi-signal Evidence (mechanical PASS, semantic PASS)
|
|
32
33
|
- **fcdb** — process-only hollow (mechanical **FAIL** after hollow-done floor; still listed for
|
|
33
34
|
historical FLAG text / offline inspection)
|
|
34
35
|
- **fcdc** — hollow-with-decoy multi-signal (mechanical PASS, semantic **FLAG** — the residual
|
|
35
36
|
attack class the mechanical floor does not claim to close)
|
|
37
|
+
- **fcdd** — B1-S artifact-less Evidence prose (mechanical PASS, semantic **FLAG** — names no
|
|
38
|
+
path/URL/command; the semantic-only catch for `ground-truth-gates.md` rule 8)
|
|
39
|
+
- **fcde** — B1-S same story with concrete artifact refs (mechanical PASS, semantic PASS)
|
|
36
40
|
|
|
37
41
|
Fixture content maps 1:1 to the failure modes already documented in `gate-rules.md`, not an
|
|
38
42
|
invented taxonomy. The v1 input set is **manifest-listed shipped fixtures only** — `eval` never
|
|
@@ -61,8 +65,10 @@ separately from a real mismatch — never silently counted as a gate pass or fai
|
|
|
61
65
|
`eval` is **opt-in and billable** — it makes real `claude -p` API calls. Zero cost when the
|
|
62
66
|
`claude` CLI isn't on `PATH` (clean skip, exit 0). When present, exactly one minimal probe call
|
|
63
67
|
is made to confirm the CLI runs headless before any real judging starts; a probe failure means
|
|
64
|
-
one billable call was made, then a clean skip.
|
|
65
|
-
|
|
68
|
+
one billable call was made, then a clean skip. The artifact manifest is 11 rows; a full
|
|
69
|
+
default batch judges the 9 heading-mapped fixtures (01 / 02 / card) × N=3 = 27 judge calls
|
|
70
|
+
+ the 1 probe = 28 calls. The two Stage 05 rows fail closed before a call. Plan accounting
|
|
71
|
+
for a complete re-record is 11 × 3 = 33 + probe.
|
|
66
72
|
|
|
67
73
|
**Measured real cost** (this machine, 2026-07-10, default/unforced model, no `--bare` available
|
|
68
74
|
under an OAuth/subscription session — see the build's spike notes): roughly **$0.30–0.37 per
|
|
@@ -78,6 +84,87 @@ fresh-judge eval is to reflect a realistic judge, not the cheapest one that pass
|
|
|
78
84
|
`--report` (below) is the free, offline way to re-read a prior batch's numbers without spending
|
|
79
85
|
anything.
|
|
80
86
|
|
|
87
|
+
**macOS without `timeout`/`gtimeout`:** live `flow.sh eval` (not `--report`, not `--replay`) refuses
|
|
88
|
+
by default. The watchdog fallback cannot bound a stuck `claude` call, so a hung judge is an
|
|
89
|
+
unbounded-billing risk. Install GNU coreutils (`gtimeout`) or export `FLOW_EVAL_UNBOUNDED=1` to
|
|
90
|
+
accept that risk for one run. `--report` and `--replay` never hit this guard. `--record` is live
|
|
91
|
+
and does.
|
|
92
|
+
|
|
93
|
+
### Cheap isolated live eval (host recipe)
|
|
94
|
+
|
|
95
|
+
Live cost is dominated by **host** context (project `CLAUDE.md`/`AGENTS.md` walk + user
|
|
96
|
+
plugins/skills/MCP), not the inlined judge prompt. Isolation is cwd + env around the
|
|
97
|
+
existing one-shot — never extra engine argv, never `--bare`, never `--model`.
|
|
98
|
+
|
|
99
|
+
`FLOW_EVAL_ISOLATED_CWD` unset → today's cwd (byte-identical). Set to an existing directory
|
|
100
|
+
→ probe and engine run in `( cd "$FLOW_EVAL_ISOLATED_CWD" && export CLAUDE_CODE_SAFE_MODE=1 &&
|
|
101
|
+
… )`. Set but not a directory → fail loud. `CLAUDE_CODE_SAFE_MODE=1` is exported only in
|
|
102
|
+
that subshell (official Claude env; keeps OAuth). Do **not** empty `HOME` or point
|
|
103
|
+
`CLAUDE_CONFIG_DIR` at the scratch dir (Linux credentials live there).
|
|
104
|
+
|
|
105
|
+
```bash
|
|
106
|
+
# Auth must be live (`claude auth login` if `claude auth status` says not logged in).
|
|
107
|
+
ISOLATED="$(mktemp -d /tmp/flow-eval-XXXXXX)" # NOT an ancestor of the flow-skill tree
|
|
108
|
+
export FLOW_EVAL_ISOLATED_CWD="$ISOLATED"
|
|
109
|
+
export FLOW_PROJECT_ROOT="/path/to/real/project" # results stay in $ROOT/.flow/
|
|
110
|
+
bash /path/to/skills/flow/runner/flow.sh eval --n 1 --fixture fcda
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
The first isolated batch is a **new baseline**, not comparable to the 2026-07-10
|
|
114
|
+
loaded-context floor ($0.30–0.37/call, ~50–60K cache-creation). A judge-model change is a
|
|
115
|
+
full re-baseline. Do not mix isolated and loaded-context batches in tripwire-2. Update the
|
|
116
|
+
measured dollars in this Cost section **after** the operator checkpoint publishes isolated
|
|
117
|
+
`total_cost_usd` + `cache_creation_input_tokens`. Replay still never counts.
|
|
118
|
+
|
|
119
|
+
Cost measurement against one captured `--output-format json` document (do not commit the
|
|
120
|
+
envelope — `session_id`/`cwd` leak):
|
|
121
|
+
|
|
122
|
+
```bash
|
|
123
|
+
printf '%s' "$raw" | python3 -c '
|
|
124
|
+
import json,sys
|
|
125
|
+
d=json.load(sys.stdin)
|
|
126
|
+
if isinstance(d,list):
|
|
127
|
+
d=next(x for x in d if isinstance(x,dict) and x.get("type")=="result")
|
|
128
|
+
u=d.get("usage") or {}
|
|
129
|
+
print("usd", d.get("total_cost_usd"))
|
|
130
|
+
print("in", u.get("input_tokens"), "out", u.get("output_tokens"))
|
|
131
|
+
print("cache_create", u.get("cache_creation_input_tokens") or u.get("cache_creation"))
|
|
132
|
+
print("cache_read", u.get("cache_read_input_tokens"))
|
|
133
|
+
print("model", d.get("model"))
|
|
134
|
+
'
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
Success signal: `cache_creation_input_tokens` collapses from ~50–60K toward the inlined
|
|
138
|
+
judge prompt size; `total_cost_usd` drops well below $0.30.
|
|
139
|
+
|
|
140
|
+
|
|
141
|
+
## Modes
|
|
142
|
+
|
|
143
|
+
| Mode | Calls | What it proves |
|
|
144
|
+
|------|-------|----------------|
|
|
145
|
+
| live (default) | Billable `claude -p` | Fresh-judge catch-rate (eval floor) |
|
|
146
|
+
| `--report` | Zero | Last complete **live** batch + advisory drift |
|
|
147
|
+
| `--record` | Same as live | Writes stripped transcripts under `eval/replay/` |
|
|
148
|
+
| `--replay` | Zero | Parse → vote → per-fixture match lines (no scorecard; no results rows) + gate-rules **hash** staleness |
|
|
149
|
+
|
|
150
|
+
`--record|--replay` are artifact-modality only (not `--stage routing|converge`, not `--report`).
|
|
151
|
+
|
|
152
|
+
**Replay is not a fresh-judge.** Gate-rules text is embedded in the live prompt; recorded
|
|
153
|
+
responses are frozen. Replay cannot test rules *effectiveness* — that stays live and billable.
|
|
154
|
+
What it can do: hard-fail when `_eval_gate_rules_sha` ≠ the recorded hash (`fixtures stale —
|
|
155
|
+
re-record live per ADR re-baseline rule`). Replay verdicts **never count toward the eval floor**.
|
|
156
|
+
|
|
157
|
+
**Refresh protocol:** edit both sides of a rules change, run a live `--record` batch (11 × `--n 3`
|
|
158
|
+
= 33 judge calls + 1 probe; 9 heading-mapped fixtures bill 27 + probe today) on a host with
|
|
159
|
+
real `timeout`/`gtimeout`, commit the stripped `eval/replay/` tree (verdict lines only — no
|
|
160
|
+
`session_id` / `cwd` envelopes) **in the same change** as the `gate-rules.md` edit. No
|
|
161
|
+
operator-recorded batch exists yet — the `eval-replay` CI job skip-with-notice until then;
|
|
162
|
+
the recorded-sha staleness check is unchanged.
|
|
163
|
+
|
|
164
|
+
**B1 escalation:** if hollow-done decoys that name no artifact/command recur in dogfood or
|
|
165
|
+
live eval after the B1-S addendum lands, escalate to full structured lineage evidence. Replay
|
|
166
|
+
verdicts never count toward that trigger.
|
|
167
|
+
|
|
81
168
|
## Running it
|
|
82
169
|
|
|
83
170
|
```
|
|
@@ -86,6 +173,8 @@ flow.sh eval --stage card # only the card-gate fixtures
|
|
|
86
173
|
flow.sh eval --fixture fcdb --n 1 # one fixture, one run (cheap smoke check)
|
|
87
174
|
flow.sh eval --timeout 180 # override the per-call timeout (default 120s)
|
|
88
175
|
flow.sh eval --report # OFFLINE: last complete batch's scorecard + drift
|
|
176
|
+
flow.sh eval --record --n 3 # LIVE: write stripped replay fixtures (billable)
|
|
177
|
+
flow.sh eval --replay --n 3 # KEYLESS: replay recorded transcripts
|
|
89
178
|
```
|
|
90
179
|
|
|
91
180
|
Exit code: `0` = every evaluated fixture majority-matched its expected verdict (or a clean
|
|
@@ -146,9 +235,10 @@ yet (the same "measure, don't assume" discipline that gated this build in the fi
|
|
|
146
235
|
platform: a signal arriving while the runner is blocked on a foreground judge call was not
|
|
147
236
|
observed to preempt that call promptly. Cleanup between fixtures/calls fires reliably; a
|
|
148
237
|
normal (uninterrupted) run always cleans up its temp files immediately.
|
|
149
|
-
- **Small fixture corpus (
|
|
150
|
-
enough for statistical confidence across every
|
|
151
|
-
corpus is a natural v1.x follow-up, not a
|
|
238
|
+
- **Small fixture corpus (eleven manifest rows, nine heading-mapped)**: enough to prove the
|
|
239
|
+
mechanism and give a real number, not enough for statistical confidence across every
|
|
240
|
+
stage/failure-mode combination. Widening the corpus is a natural v1.x follow-up, not a
|
|
241
|
+
v1 blocker.
|
|
152
242
|
|
|
153
243
|
## Failure modes and postmortem (v0.21)
|
|
154
244
|
|
|
@@ -174,8 +264,8 @@ window; the mechanism was never confirmed at the time (rate-limit vs hook conten
|
|
|
174
264
|
4. **Circuit breaker.** The FIRST evaluated fixture coming back UNRELIABLE (`invalid_count*3 > n`)
|
|
175
265
|
aborts the batch with a distinct nonzero exit code (2) and NO `done` trailer written — the
|
|
176
266
|
junk batch is invisible to `--report`/drift, so it cannot poison the baseline. `--keep-going`
|
|
177
|
-
forces the full batch; document worst case (≤ N_fixtures × n × 2 + 1 probe ≈
|
|
178
|
-
default
|
|
267
|
+
forces the full batch; document worst case (≤ N_fixtures × n × 2 + 1 probe ≈ 55 calls at
|
|
268
|
+
default 9 heading-mapped × 3).
|
|
179
269
|
|
|
180
270
|
### Rate-limit visibility (advisory, best-effort)
|
|
181
271
|
|
|
@@ -182,6 +182,9 @@ evidence non-empty.
|
|
|
182
182
|
- **Is `## Evidence` real world-state** — a clickable URL, real curl output, a DB row — and
|
|
183
183
|
NOT "tests pass" / "code merged" / "deployed successfully"? Merge != shipped: the proof is
|
|
184
184
|
the live surface changing, verified as a user.
|
|
185
|
+
- **Does every `## Evidence` item name its artifact (path/URL) or the command that produced
|
|
186
|
+
it?** Plausible prose that names neither is still hollow even when the mechanical floor
|
|
187
|
+
passes (`ground-truth-gates.md` rule 8).
|
|
185
188
|
- **If this card fixes a bug/regression**, the evidence includes the red→green proof that the
|
|
186
189
|
new test was actually tied to the bug (`ground-truth-gates.md` → "Bug-fix cards"), not just a
|
|
187
190
|
green run.
|
|
@@ -40,6 +40,13 @@ and used only to add color, never to pass a gate.
|
|
|
40
40
|
7. **Hand-edit `status: done` is not trusted for deps** until Evidence **and** Verify boxes
|
|
41
41
|
pass the same floor (`ready` re-validates Evidence + unchecked Verify + `[FILL]`; graph
|
|
42
42
|
executor mirrors).
|
|
43
|
+
8. **Every done-evidence item names its artifact or command.** Each `## Evidence` item must
|
|
44
|
+
name the path/URL of the artifact or the command that produced it. Plausible prose that
|
|
45
|
+
names neither is hollow even when the mechanical multi-signal floor passes — the card
|
|
46
|
+
gate's semantic challenge catches it (`gate-rules.md` → Card gate). Old prose-only
|
|
47
|
+
evidence still passes `flow.sh check` (the floor is unchanged). **Escalate to full
|
|
48
|
+
structured lineage evidence (B1)** only if hollow-done decoys of this class recur in
|
|
49
|
+
dogfood or live eval after this addendum lands.
|
|
43
50
|
|
|
44
51
|
## Bug-fix cards: prove the test was tied to the bug
|
|
45
52
|
When a card's job is fixing a bug/regression (not new behavior), a passing test is not yet
|
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
# Host-agnostic parallel occupancy
|
|
2
|
+
|
|
3
|
+
Flow requires **no named multiplexer**. Parallel cards are a ready-gate +
|
|
4
|
+
worktrees + host-owned placement + world-state fan-in. Absence of Herdr, tmux,
|
|
5
|
+
in-process `Task`, or any other host **never fails a gate** and never changes
|
|
6
|
+
`flow.sh` exit.
|
|
7
|
+
|
|
8
|
+
Herdr and tmux appear below only as **examples, never as a dependency** or a
|
|
9
|
+
first-class adapter.
|
|
10
|
+
|
|
11
|
+
## Degrade ladder
|
|
12
|
+
|
|
13
|
+
| Rung | Primitive already in the process | Who binds occupants | Who waits |
|
|
14
|
+
|---|---|---|---|
|
|
15
|
+
| **0** | In-process subagents / `Task` | Hosting agent; cwd = the card's worktree | Host `Task` join |
|
|
16
|
+
| **1** | Operator extra terminals / tabs | Human pastes the print-enter block | Human |
|
|
17
|
+
| **2** | Process is **already inside** a mux (`HERDR_ENV=1` / `TMUX` set) | That mux's **own** skill/CLI; flow does not wrap or detect it | That mux's own wait |
|
|
18
|
+
| **3** | None of the above, or `/flow ready` overlap | Serial: one card | Hosting agent, one at a time |
|
|
19
|
+
|
|
20
|
+
Rung 2 is opt-in by the mux's own environment injection, never by `PATH`.
|
|
21
|
+
Inside-mux probe is the mux's own env (`HERDR_ENV=1`, `TMUX`), never
|
|
22
|
+
`command -v herdr` from outside (a CLI can talk to a same-uid socket without
|
|
23
|
+
`HERDR_ENV`). Absence of rungs 0–2 is the default path, not a degrade footnote.
|
|
24
|
+
|
|
25
|
+
## Gate parity
|
|
26
|
+
|
|
27
|
+
Host `idle` / `done` / `blocked` is **not a card pass**. `flow.sh check` still
|
|
28
|
+
judges, and it must run **in the card's worktree** (parent-tree check of main
|
|
29
|
+
is a false fail/pass). Screen output is not evidence.
|
|
30
|
+
|
|
31
|
+
Keep the four "blocked" senses distinct:
|
|
32
|
+
|
|
33
|
+
| Sense | Meaning | Home |
|
|
34
|
+
|---|---|---|
|
|
35
|
+
| **host-blocked** | mux/pane approval chrome or occupant stuck | the host |
|
|
36
|
+
| **subagent-BLOCKED** | scoped subagent status protocol | `agent-detection.md` |
|
|
37
|
+
| **ready-blocked** | deps unmet or allowed-files overlap | `flow.sh ready` |
|
|
38
|
+
| **Tier-C security halt** | security-class exposure; operator DEBT | `auto-run.md` |
|
|
39
|
+
|
|
40
|
+
Mixing them auto-answers a payments card or auto-halts a yes/no prompt.
|
|
41
|
+
|
|
42
|
+
## File-is-the-wait
|
|
43
|
+
|
|
44
|
+
The **HOST waits** (Task join / human / the mux's own wait). `flow.sh` **never waits on or polls** another agent. Fan-in is world-state: re-invoke `check` in
|
|
45
|
+
the card worktree after the host's wait returns.
|
|
46
|
+
|
|
47
|
+
## Forbidden (ADR-0001; not a nit)
|
|
48
|
+
|
|
49
|
+
`flow.sh` must not exec host control CLIs (`herdr agent start`, `herdr agent wait`, `herdr agent prompt`, `herdr server stop`, `tmux send-keys`, and equivalents).
|
|
50
|
+
Forbidden by name: `mux-up`, `mux-run-wave`, `mux-proxy`. No screen output as
|
|
51
|
+
evidence. No hook install into `~/.claude` or `~/.omp`. No long-lived socket subscribe (`events.subscribe`). `flow.sh` never spawns agents; placement is print-enter or the host's own Task/mux.
|
|
52
|
+
|
|
53
|
+
Own-runtime / own-PTY / own-wait is a successor ADR plus a separate product,
|
|
54
|
+
not a card in this skill.
|
|
@@ -69,6 +69,8 @@ HARNESS_PY="$SCRIPT_DIR/../harness/flow_harness.py"
|
|
|
69
69
|
# so an overridden manifest can only ever point at real shipped fixtures under the real EVAL_DIR.
|
|
70
70
|
EVAL_DIR="$SCRIPT_DIR/../eval"
|
|
71
71
|
EVAL_MANIFEST="${FLOW_EVAL_MANIFEST:-$EVAL_DIR/manifest.tsv}"
|
|
72
|
+
# FLOW_EVAL_REPLAY_DIR is a TEST-ONLY override (synthetic harness-built transcripts).
|
|
73
|
+
EVAL_REPLAY_DIR="${FLOW_EVAL_REPLAY_DIR:-$EVAL_DIR/replay}"
|
|
72
74
|
GATE_RULES_FILE="$SCRIPT_DIR/../references/gate-rules.md"
|
|
73
75
|
|
|
74
76
|
# Routing eval (v0.22 concierge routing judge - a SEPARATE modality from the artifact-vs-
|
|
@@ -2300,6 +2302,14 @@ cmd_doctor() {
|
|
|
2300
2302
|
command -v cargo >/dev/null 2>&1 && echo " cargo: ok (optional Rust harness power-path)" || true
|
|
2301
2303
|
[ -d "$TEMPLATE_DIR" ] && echo " templates: ok" || { echo " templates: MISSING ($TEMPLATE_DIR)"; ok=0; }
|
|
2302
2304
|
[ -f "$HARNESS_PY" ] && echo " harness: present ($HARNESS_PY)" || echo " harness: absent (durable layer off)"
|
|
2305
|
+
if command -v timeout >/dev/null 2>&1 || command -v gtimeout >/dev/null 2>&1; then
|
|
2306
|
+
echo " timeout: full (timeout|gtimeout present)"
|
|
2307
|
+
else
|
|
2308
|
+
case "$(uname -s 2>/dev/null)" in
|
|
2309
|
+
Darwin) echo " timeout: absent (no timeout/gtimeout; live eval REFUSED)" ;;
|
|
2310
|
+
*) echo " timeout: partial (watchdog fallback; live eval not GNU-bounded)" ;;
|
|
2311
|
+
esac
|
|
2312
|
+
fi
|
|
2303
2313
|
echo
|
|
2304
2314
|
if [ "$ok" -eq 1 ]; then
|
|
2305
2315
|
echo "READY. Gate engine works$([ -n "$py" ] && echo '; durable layer on.' || echo '; durable layer off (no python).')"
|
|
@@ -3197,6 +3207,25 @@ _eval_engine_run() { # $1=promptfile $2=timeout-seconds -> stdout=raw json; retu
|
|
|
3197
3207
|
# command string above. Empirical: adding a `2> '$errfile'` inside the timed cmd breaks the
|
|
3198
3208
|
# watchdog-fallback path in `_run_with_timeout` on the timeout-less-PATH lane (macOS DEBT lane).
|
|
3199
3209
|
# Outer-scope redirection is exactly the same capture at zero cost to the timeout contract.
|
|
3210
|
+
# Opt-in isolated cwd for live eval (Approach D). Unset = today's cwd (byte-identical
|
|
3211
|
+
# call path). Set but not a directory = fail loud. Isolation is cd+env around existing
|
|
3212
|
+
# probe/engine calls — never argv, never a second engine. CLAUDE_CODE_SAFE_MODE=1 is
|
|
3213
|
+
# exported ONLY in the isolated subshell (official Claude env; keeps OAuth; blast radius
|
|
3214
|
+
# is the judge child). Do not invent a flow alias; do not set HOME or CLAUDE_CONFIG_DIR.
|
|
3215
|
+
_eval_isolated_guard() {
|
|
3216
|
+
[ -z "${FLOW_EVAL_ISOLATED_CWD:-}" ] && return 0
|
|
3217
|
+
[ -d "$FLOW_EVAL_ISOLATED_CWD" ] && return 0
|
|
3218
|
+
echo "FAIL: FLOW_EVAL_ISOLATED_CWD is set but is not a directory: $FLOW_EVAL_ISOLATED_CWD"
|
|
3219
|
+
return 1
|
|
3220
|
+
}
|
|
3221
|
+
_eval_isolated_run() {
|
|
3222
|
+
if [ -n "${FLOW_EVAL_ISOLATED_CWD:-}" ]; then
|
|
3223
|
+
( cd "$FLOW_EVAL_ISOLATED_CWD" && export CLAUDE_CODE_SAFE_MODE=1 && "$@" )
|
|
3224
|
+
else
|
|
3225
|
+
"$@"
|
|
3226
|
+
fi
|
|
3227
|
+
}
|
|
3228
|
+
|
|
3200
3229
|
|
|
3201
3230
|
# Build the judge prompt for one fixture. Returns 1 (writes nothing usable) if the gate-rules.md
|
|
3202
3231
|
# section extraction came back empty.
|
|
@@ -3289,6 +3318,13 @@ _eval_gate_rules_sha() {
|
|
|
3289
3318
|
tr -d '\r' < "$GATE_RULES_FILE" | cksum | awk '{print $1}'
|
|
3290
3319
|
}
|
|
3291
3320
|
|
|
3321
|
+
# CRLF-normalized assembler-output hash (same recipe as _eval_gate_rules_sha). Covers
|
|
3322
|
+
# prompt-file bytes only — not host CLAUDE.md. $1=prompt file.
|
|
3323
|
+
_eval_prompt_sha() {
|
|
3324
|
+
[ -f "$1" ] || { printf 'unknown'; return; }
|
|
3325
|
+
tr -d '\r' < "$1" | cksum | awk '{print $1}'
|
|
3326
|
+
}
|
|
3327
|
+
|
|
3292
3328
|
# `claude --version` is the CLI version (e.g. "2.1.201 (Claude Code)"), NOT a model id - kept
|
|
3293
3329
|
# as its own axis distinct from `_eval_parse_model` (red-team finding #4: 3 reviewers
|
|
3294
3330
|
# independently confirmed conflating the two is wrong). `< /dev/null`: this and every other
|
|
@@ -3302,6 +3338,48 @@ _eval_cli_version() {
|
|
|
3302
3338
|
[ -n "$v" ] && printf '%s' "$v" || printf 'unknown'
|
|
3303
3339
|
}
|
|
3304
3340
|
|
|
3341
|
+
# Replay/record helpers. Do not live inside _eval_engine_run. Meta is key=value, one per line.
|
|
3342
|
+
_eval_replay_meta_get() { # $1=key -> value (empty + return 1 if missing)
|
|
3343
|
+
local key="$1" line
|
|
3344
|
+
[ -f "$EVAL_REPLAY_DIR/meta" ] || return 1
|
|
3345
|
+
while IFS= read -r line || [ -n "$line" ]; do
|
|
3346
|
+
line="${line%$'\r'}"
|
|
3347
|
+
case "$line" in
|
|
3348
|
+
"${key}="*) printf '%s' "${line#${key}=}"; return 0 ;;
|
|
3349
|
+
esac
|
|
3350
|
+
done < "$EVAL_REPLAY_DIR/meta"
|
|
3351
|
+
return 1
|
|
3352
|
+
}
|
|
3353
|
+
|
|
3354
|
+
_eval_replay_write_meta() { # $1=nonce $2=gate_rules_sha $3=model $4=n [$5=prompt_sha.<fid>= lines]
|
|
3355
|
+
mkdir -p "$EVAL_REPLAY_DIR" || return 1
|
|
3356
|
+
printf 'nonce=%s\ngate_rules_sha=%s\nmodel=%s\nn=%s\n' "$1" "$2" "$3" "$4" > "$EVAL_REPLAY_DIR/meta" || return 1
|
|
3357
|
+
[ -n "${5:-}" ] && printf '%s' "$5" >> "$EVAL_REPLAY_DIR/meta"
|
|
3358
|
+
}
|
|
3359
|
+
|
|
3360
|
+
# Persist one stripped verdict line only — never a JSON envelope (session_id/cwd stay out).
|
|
3361
|
+
_eval_fid_safe() { # $1=fid -> filename-safe token (no path separators)
|
|
3362
|
+
local s
|
|
3363
|
+
s="$(printf '%s' "$1" | tr -c 'A-Za-z0-9' '-' | sed -E 's/-+/-/g; s/^-//; s/-$//')"
|
|
3364
|
+
[ -n "$s" ] || s="_"
|
|
3365
|
+
printf '%s' "$s"
|
|
3366
|
+
}
|
|
3367
|
+
|
|
3368
|
+
_eval_record_vote() { # $1=fid $2=vote-i $3=nonce $4=FLAG|PASS|INVALID
|
|
3369
|
+
local dest="$EVAL_REPLAY_DIR/$(_eval_fid_safe "$1")"
|
|
3370
|
+
mkdir -p "$dest" || return 1
|
|
3371
|
+
printf 'GATE-EVAL-%s: %s\n' "$3" "$4" > "$dest/$2.txt"
|
|
3372
|
+
}
|
|
3373
|
+
|
|
3374
|
+
_eval_replay_vote() { # $1=fid $2=vote-i -> transcript on stdout; return 1 if missing
|
|
3375
|
+
local f="$EVAL_REPLAY_DIR/$(_eval_fid_safe "$1")/$2.txt"
|
|
3376
|
+
if [ ! -f "$f" ]; then
|
|
3377
|
+
echo "eval --replay: missing fixture $1 vote $2 at $f" >&2
|
|
3378
|
+
return 1
|
|
3379
|
+
fi
|
|
3380
|
+
cat "$f"
|
|
3381
|
+
}
|
|
3382
|
+
|
|
3305
3383
|
# Batch header/trailer: the completeness signal `--report`/drift filter on is "start present +
|
|
3306
3384
|
# done present + n_written >= n_expected". Two paths lead to a MISSING trailer (correctly
|
|
3307
3385
|
# incomplete): (a) an interrupted batch (INT/TERM trap exits before this line, per the eval
|
|
@@ -3323,17 +3401,18 @@ _eval_emit_batch_marker() { # $1=run_id $2=start|done $3=n_expected-or-n_written
|
|
|
3323
3401
|
# cli_version/flow_version/gate_rules_sha are passed in (computed ONCE per batch by the caller,
|
|
3324
3402
|
# not per fixture row) - they never change mid-batch, and recomputing per-row was both wasteful
|
|
3325
3403
|
# (an extra subprocess per fixture) and the original vector for the stdin-consumption bug above.
|
|
3326
|
-
_eval_emit_result() { # $1..$14 as before + $15=retries $16=rate_limited
|
|
3404
|
+
_eval_emit_result() { # $1..$14 as before + $15=retries $16=rate_limited $17=prompt_sha $18=timed_out
|
|
3327
3405
|
local run_id="$1" fid="$2" stage="$3" expected="$4" verdict="$5" match="$6" flag="$7" pass="$8" invalid="$9" n="${10}" model="${11}" cliv="${12}" flowv="${13}" grsha="${14}"
|
|
3328
|
-
local retries="${15:-0}" ratel="${16:-false}"
|
|
3406
|
+
local retries="${15:-0}" ratel="${16:-false}" psha="${17:-}" timedout="${18:-false}"
|
|
3329
3407
|
case "$retries" in ''|*[!0-9]*) retries=0 ;; esac
|
|
3330
3408
|
case "$ratel" in true|false) : ;; *) ratel=false ;; esac
|
|
3409
|
+
case "$timedout" in true|false) : ;; *) timedout=false ;; esac
|
|
3331
3410
|
mkdir -p "$LOG_DIR" 2>/dev/null || true
|
|
3332
|
-
printf '{"ts":"%s","epoch_s":%s,"run_id":"%s","fixture":"%s","stage":"%s","expected":"%s","verdict":"%s","match":"%s","votes":{"flag":%s,"pass":%s,"invalid":%s},"n":%s,"cli_version":"%s","model":"%s","flow_version":"%s","gate_rules_sha":"%s","retries":%s,"rate_limited":%s}\n' \
|
|
3411
|
+
printf '{"ts":"%s","epoch_s":%s,"run_id":"%s","fixture":"%s","stage":"%s","expected":"%s","verdict":"%s","match":"%s","votes":{"flag":%s,"pass":%s,"invalid":%s},"n":%s,"cli_version":"%s","model":"%s","flow_version":"%s","gate_rules_sha":"%s","retries":%s,"rate_limited":%s,"prompt_sha":"%s","timed_out":%s}\n' \
|
|
3333
3412
|
"$(date -u +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || echo unknown)" "$(_now)" "$(_json_str "$run_id")" "$(_json_str "$fid")" \
|
|
3334
3413
|
"$(_json_str "$stage")" "$(_json_str "$expected")" "$(_json_str "$verdict")" "$(_json_str "$match")" \
|
|
3335
3414
|
"$flag" "$pass" "$invalid" "$n" "$(_json_str "$cliv")" "$(_json_str "$model")" \
|
|
3336
|
-
"$(_json_str "$flowv")" "$(_json_str "$grsha")" "$retries" "$ratel" >> "$EVAL_RESULTS_FILE" 2>/dev/null || true
|
|
3415
|
+
"$(_json_str "$flowv")" "$(_json_str "$grsha")" "$retries" "$ratel" "$(_json_str "$psha")" "$timedout" >> "$EVAL_RESULTS_FILE" 2>/dev/null || true
|
|
3337
3416
|
}
|
|
3338
3417
|
|
|
3339
3418
|
# Strip the noisy `system`/`init` envelope from a raw --output-format json line before persisting
|
|
@@ -3617,17 +3696,18 @@ _eval_routing_parse_action() {
|
|
|
3617
3696
|
[ -n "$action" ] && printf '%s' "$action" || echo INVALID
|
|
3618
3697
|
}
|
|
3619
3698
|
|
|
3620
|
-
_eval_routing_emit_result() { # $1=run_id $2=fid $3=expected $4=verdict $5=match $6..$9=votes
|
|
3699
|
+
_eval_routing_emit_result() { # $1=run_id $2=fid $3=expected $4=verdict $5=match $6..$9=votes $10=model $11=cliv $12=flowv $13=rulessha $14=retries $15=rate_limited $16=prompt_sha $17=timed_out
|
|
3621
3700
|
local run_id="$1" fid="$2" expected="$3" verdict="$4" match="$5" hit="$6" miss="$7" invalid="$8" n="$9"
|
|
3622
|
-
local model="${10}" cliv="${11}" flowv="${12}" rulessha="${13}" retries="${14:-0}" ratel="${15:-false}"
|
|
3701
|
+
local model="${10}" cliv="${11}" flowv="${12}" rulessha="${13}" retries="${14:-0}" ratel="${15:-false}" psha="${16:-}" timedout="${17:-false}"
|
|
3623
3702
|
case "$retries" in ''|*[!0-9]*) retries=0 ;; esac
|
|
3624
3703
|
case "$ratel" in true|false) : ;; *) ratel=false ;; esac
|
|
3704
|
+
case "$timedout" in true|false) : ;; *) timedout=false ;; esac
|
|
3625
3705
|
mkdir -p "$LOG_DIR" 2>/dev/null || true
|
|
3626
|
-
printf '{"ts":"%s","epoch_s":%s,"run_id":"%s","fixture":"%s","expected":"%s","verdict":"%s","match":"%s","votes":{"hit":%s,"miss":%s,"invalid":%s},"n":%s,"cli_version":"%s","model":"%s","flow_version":"%s","rules_sha":"%s","retries":%s,"rate_limited":%s}\n' \
|
|
3706
|
+
printf '{"ts":"%s","epoch_s":%s,"run_id":"%s","fixture":"%s","expected":"%s","verdict":"%s","match":"%s","votes":{"hit":%s,"miss":%s,"invalid":%s},"n":%s,"cli_version":"%s","model":"%s","flow_version":"%s","rules_sha":"%s","retries":%s,"rate_limited":%s,"prompt_sha":"%s","timed_out":%s}\n' \
|
|
3627
3707
|
"$(date -u +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || echo unknown)" "$(_now)" "$(_json_str "$run_id")" "$(_json_str "$fid")" \
|
|
3628
3708
|
"$(_json_str "$expected")" "$(_json_str "$verdict")" "$(_json_str "$match")" \
|
|
3629
3709
|
"$hit" "$miss" "$invalid" "$n" "$(_json_str "$cliv")" "$(_json_str "$model")" \
|
|
3630
|
-
"$(_json_str "$flowv")" "$(_json_str "$rulessha")" "$retries" "$ratel" >> "$EVAL_ROUTING_RESULTS_FILE" 2>/dev/null || true
|
|
3710
|
+
"$(_json_str "$flowv")" "$(_json_str "$rulessha")" "$retries" "$ratel" "$(_json_str "$psha")" "$timedout" >> "$EVAL_ROUTING_RESULTS_FILE" 2>/dev/null || true
|
|
3631
3711
|
}
|
|
3632
3712
|
|
|
3633
3713
|
_eval_routing_complete_run_ids() {
|
|
@@ -3727,7 +3807,9 @@ function cmd_eval_routing { # same arg shape as cmd_eval, called only when --sta
|
|
|
3727
3807
|
return 1
|
|
3728
3808
|
fi
|
|
3729
3809
|
|
|
3730
|
-
|
|
3810
|
+
_eval_isolated_guard || return 1
|
|
3811
|
+
|
|
3812
|
+
local probe; probe="$(_eval_isolated_run _eval_probe)"
|
|
3731
3813
|
case "$probe" in
|
|
3732
3814
|
absent)
|
|
3733
3815
|
echo "SKIP: 'claude' CLI not found on PATH - routing eval needs it to run the LLM judge. Zero calls made."
|
|
@@ -3795,14 +3877,18 @@ function cmd_eval_routing { # same arg shape as cmd_eval, called only when --sta
|
|
|
3795
3877
|
continue
|
|
3796
3878
|
fi
|
|
3797
3879
|
|
|
3880
|
+
local psha
|
|
3881
|
+
psha="$(_eval_prompt_sha "$promptfile")"
|
|
3882
|
+
|
|
3798
3883
|
total_evaluated=$((total_evaluated + 1))
|
|
3799
|
-
local hit_count=0 miss_count=0 invalid_count=0 i=1 model="unknown" retries_sum=0 vote_rate_limited=false
|
|
3884
|
+
local hit_count=0 miss_count=0 invalid_count=0 i=1 model="unknown" retries_sum=0 vote_rate_limited=false vote_timed_out=false
|
|
3800
3885
|
while [ "$i" -le "$n" ]; do
|
|
3801
3886
|
local raw rc v raw1="" rc1=0
|
|
3802
|
-
raw="$(_eval_engine_run "$promptfile" "$timeout" 2>/dev/null)"; rc=$?
|
|
3887
|
+
raw="$(_eval_isolated_run _eval_engine_run "$promptfile" "$timeout" 2>/dev/null)"; rc=$?
|
|
3803
3888
|
raw1="$raw"; rc1="$rc"
|
|
3804
3889
|
if [ "$rc" -eq 124 ]; then
|
|
3805
3890
|
v=INVALID
|
|
3891
|
+
vote_timed_out=true
|
|
3806
3892
|
else
|
|
3807
3893
|
v="$(_eval_routing_parse_action "$raw" "$nonce")"
|
|
3808
3894
|
[ "$model" = "unknown" ] && model="$(_eval_parse_model "$raw")"
|
|
@@ -3814,9 +3900,9 @@ function cmd_eval_routing { # same arg shape as cmd_eval, called only when --sta
|
|
|
3814
3900
|
echo " $fid: retrying vote $i after ${backoff}s (parse-INVALID on attempt 1)"
|
|
3815
3901
|
sleep "$backoff" 2>/dev/null || true
|
|
3816
3902
|
fi
|
|
3817
|
-
raw="$(_eval_engine_run "$promptfile" "$timeout" 2>/dev/null)"; rc=$?
|
|
3903
|
+
raw="$(_eval_isolated_run _eval_engine_run "$promptfile" "$timeout" 2>/dev/null)"; rc=$?
|
|
3818
3904
|
retries_sum=$((retries_sum + 1))
|
|
3819
|
-
if [ "$rc" -eq 124 ]; then v=INVALID; else v="$(_eval_routing_parse_action "$raw" "$nonce")"; fi
|
|
3905
|
+
if [ "$rc" -eq 124 ]; then v=INVALID; vote_timed_out=true; else v="$(_eval_routing_parse_action "$raw" "$nonce")"; fi
|
|
3820
3906
|
[ "$model" = "unknown" ] && [ "$rc" -ne 124 ] && model="$(_eval_parse_model "$raw")"
|
|
3821
3907
|
local rl2; rl2="$(_eval_parse_rate_limited "$raw")"
|
|
3822
3908
|
[ "$rl2" = "true" ] && vote_rate_limited=true
|
|
@@ -3854,7 +3940,7 @@ function cmd_eval_routing { # same arg shape as cmd_eval, called only when --sta
|
|
|
3854
3940
|
echo " $fid: MISS - expected '$fexpected' (hit=$hit_count miss=$miss_count invalid=$invalid_count)"
|
|
3855
3941
|
fi
|
|
3856
3942
|
|
|
3857
|
-
_eval_routing_emit_result "$nonce" "$fid" "$fexpected" "$verdict" "$match" "$hit_count" "$miss_count" "$invalid_count" "$n" "$model" "$batch_cliv" "$batch_flowv" "$batch_rulessha" "$retries_sum" "$vote_rate_limited"
|
|
3943
|
+
_eval_routing_emit_result "$nonce" "$fid" "$fexpected" "$verdict" "$match" "$hit_count" "$miss_count" "$invalid_count" "$n" "$model" "$batch_cliv" "$batch_flowv" "$batch_rulessha" "$retries_sum" "$vote_rate_limited" "$psha" "$vote_timed_out"
|
|
3858
3944
|
n_written=$((n_written + 1))
|
|
3859
3945
|
|
|
3860
3946
|
if [ "$total_evaluated" -eq 1 ] && [ "$verdict" = "UNRELIABLE" ] && [ "$keep_going" -eq 0 ]; then
|
|
@@ -3962,15 +4048,16 @@ _eval_converge_parse_verdict() { # $1=raw $2=nonce -> GAP|CONVERGED|INVALID
|
|
|
3962
4048
|
|
|
3963
4049
|
_eval_converge_emit_result() { # same field shape as _eval_routing_emit_result
|
|
3964
4050
|
local run_id="$1" fid="$2" expected="$3" verdict="$4" match="$5" hit="$6" miss="$7" invalid="$8" n="$9"
|
|
3965
|
-
local model="${10}" cliv="${11}" flowv="${12}" rulessha="${13}" retries="${14:-0}" ratel="${15:-false}"
|
|
4051
|
+
local model="${10}" cliv="${11}" flowv="${12}" rulessha="${13}" retries="${14:-0}" ratel="${15:-false}" psha="${16:-}" timedout="${17:-false}"
|
|
3966
4052
|
case "$retries" in ''|*[!0-9]*) retries=0 ;; esac
|
|
3967
4053
|
case "$ratel" in true|false) : ;; *) ratel=false ;; esac
|
|
4054
|
+
case "$timedout" in true|false) : ;; *) timedout=false ;; esac
|
|
3968
4055
|
mkdir -p "$LOG_DIR" 2>/dev/null || true
|
|
3969
|
-
printf '{"ts":"%s","epoch_s":%s,"run_id":"%s","fixture":"%s","expected":"%s","verdict":"%s","match":"%s","votes":{"hit":%s,"miss":%s,"invalid":%s},"n":%s,"cli_version":"%s","model":"%s","flow_version":"%s","rules_sha":"%s","retries":%s,"rate_limited":%s}\n' \
|
|
4056
|
+
printf '{"ts":"%s","epoch_s":%s,"run_id":"%s","fixture":"%s","expected":"%s","verdict":"%s","match":"%s","votes":{"hit":%s,"miss":%s,"invalid":%s},"n":%s,"cli_version":"%s","model":"%s","flow_version":"%s","rules_sha":"%s","retries":%s,"rate_limited":%s,"prompt_sha":"%s","timed_out":%s}\n' \
|
|
3970
4057
|
"$(date -u +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || echo unknown)" "$(_now)" "$(_json_str "$run_id")" "$(_json_str "$fid")" \
|
|
3971
4058
|
"$(_json_str "$expected")" "$(_json_str "$verdict")" "$(_json_str "$match")" \
|
|
3972
4059
|
"$hit" "$miss" "$invalid" "$n" "$(_json_str "$cliv")" "$(_json_str "$model")" \
|
|
3973
|
-
"$(_json_str "$flowv")" "$(_json_str "$rulessha")" "$retries" "$ratel" >> "$EVAL_CONVERGE_RESULTS_FILE" 2>/dev/null || true
|
|
4060
|
+
"$(_json_str "$flowv")" "$(_json_str "$rulessha")" "$retries" "$ratel" "$(_json_str "$psha")" "$timedout" >> "$EVAL_CONVERGE_RESULTS_FILE" 2>/dev/null || true
|
|
3974
4061
|
}
|
|
3975
4062
|
|
|
3976
4063
|
_eval_converge_emit_batch_marker() { # $1=run_id $2=start|done $3=n
|
|
@@ -4032,7 +4119,9 @@ function cmd_eval_converge { # same arg shape as cmd_eval, called only when --st
|
|
|
4032
4119
|
return 1
|
|
4033
4120
|
fi
|
|
4034
4121
|
|
|
4035
|
-
|
|
4122
|
+
_eval_isolated_guard || return 1
|
|
4123
|
+
|
|
4124
|
+
local probe; probe="$(_eval_isolated_run _eval_probe)"
|
|
4036
4125
|
case "$probe" in
|
|
4037
4126
|
absent) echo "SKIP: 'claude' CLI not found on PATH - converge eval needs it to run the LLM judge. Zero calls made."; return 0 ;;
|
|
4038
4127
|
fail) echo "SKIP: 'claude' CLI present but the sentinel probe did not come back clean (one minimal probe call was made). Retry once 'claude -p' runs headless."; return 0 ;;
|
|
@@ -4080,17 +4169,20 @@ function cmd_eval_converge { # same arg shape as cmd_eval, called only when --st
|
|
|
4080
4169
|
echo " $fid: FAIL - could not build converge prompt (converge.md/repo-dir/challenge missing)"; total_miss=$((total_miss + 1)); rm -rf "$rundir" 2>/dev/null; continue
|
|
4081
4170
|
fi
|
|
4082
4171
|
|
|
4172
|
+
local psha
|
|
4173
|
+
psha="$(_eval_prompt_sha "$promptfile")"
|
|
4174
|
+
|
|
4083
4175
|
total_evaluated=$((total_evaluated + 1))
|
|
4084
|
-
local hit_count=0 miss_count=0 invalid_count=0 i=1 model="unknown" retries_sum=0 vote_rate_limited=false
|
|
4176
|
+
local hit_count=0 miss_count=0 invalid_count=0 i=1 model="unknown" retries_sum=0 vote_rate_limited=false vote_timed_out=false
|
|
4085
4177
|
while [ "$i" -le "$n" ]; do
|
|
4086
4178
|
local raw rc v raw1="" rc1=0
|
|
4087
|
-
raw="$(_eval_engine_run "$promptfile" "$timeout" 2>/dev/null)"; rc=$?; raw1="$raw"; rc1="$rc"
|
|
4088
|
-
if [ "$rc" -eq 124 ]; then v=INVALID; else v="$(_eval_converge_parse_verdict "$raw" "$nonce")"; [ "$model" = "unknown" ] && model="$(_eval_parse_model "$raw")"; fi
|
|
4179
|
+
raw="$(_eval_isolated_run _eval_engine_run "$promptfile" "$timeout" 2>/dev/null)"; rc=$?; raw1="$raw"; rc1="$rc"
|
|
4180
|
+
if [ "$rc" -eq 124 ]; then v=INVALID; vote_timed_out=true; else v="$(_eval_converge_parse_verdict "$raw" "$nonce")"; [ "$model" = "unknown" ] && model="$(_eval_parse_model "$raw")"; fi
|
|
4089
4181
|
local rl1; rl1="$(_eval_parse_rate_limited "$raw1")"; [ "$rl1" = "true" ] && vote_rate_limited=true
|
|
4090
4182
|
if [ "$v" = "INVALID" ] && [ "$rl1" != "true" ] && [ "$rc1" -ne 124 ]; then
|
|
4091
4183
|
if [ "$backoff" -gt 0 ]; then echo " $fid: retrying vote $i after ${backoff}s (parse-INVALID on attempt 1)"; sleep "$backoff" 2>/dev/null || true; fi
|
|
4092
|
-
raw="$(_eval_engine_run "$promptfile" "$timeout" 2>/dev/null)"; rc=$?; retries_sum=$((retries_sum + 1))
|
|
4093
|
-
if [ "$rc" -eq 124 ]; then v=INVALID; else v="$(_eval_converge_parse_verdict "$raw" "$nonce")"; fi
|
|
4184
|
+
raw="$(_eval_isolated_run _eval_engine_run "$promptfile" "$timeout" 2>/dev/null)"; rc=$?; retries_sum=$((retries_sum + 1))
|
|
4185
|
+
if [ "$rc" -eq 124 ]; then v=INVALID; vote_timed_out=true; else v="$(_eval_converge_parse_verdict "$raw" "$nonce")"; fi
|
|
4094
4186
|
[ "$model" = "unknown" ] && [ "$rc" -ne 124 ] && model="$(_eval_parse_model "$raw")"
|
|
4095
4187
|
local rl2; rl2="$(_eval_parse_rate_limited "$raw")"; [ "$rl2" = "true" ] && vote_rate_limited=true
|
|
4096
4188
|
fi
|
|
@@ -4115,7 +4207,7 @@ function cmd_eval_converge { # same arg shape as cmd_eval, called only when --st
|
|
|
4115
4207
|
total_miss=$((total_miss + 1)); echo " $fid: MISS - expected '$fexpected' (hit=$hit_count miss=$miss_count invalid=$invalid_count)"
|
|
4116
4208
|
fi
|
|
4117
4209
|
|
|
4118
|
-
_eval_converge_emit_result "$nonce" "$fid" "$fexpected" "$verdict" "$match" "$hit_count" "$miss_count" "$invalid_count" "$n" "$model" "$batch_cliv" "$batch_flowv" "$batch_rulessha" "$retries_sum" "$vote_rate_limited"
|
|
4210
|
+
_eval_converge_emit_result "$nonce" "$fid" "$fexpected" "$verdict" "$match" "$hit_count" "$miss_count" "$invalid_count" "$n" "$model" "$batch_cliv" "$batch_flowv" "$batch_rulessha" "$retries_sum" "$vote_rate_limited" "$psha" "$vote_timed_out"
|
|
4119
4211
|
n_written=$((n_written + 1))
|
|
4120
4212
|
|
|
4121
4213
|
if [ "$total_evaluated" -eq 1 ] && [ "$verdict" = "UNRELIABLE" ] && [ "$keep_going" -eq 0 ]; then
|
|
@@ -4138,6 +4230,30 @@ function cmd_eval_converge { # same arg shape as cmd_eval, called only when --st
|
|
|
4138
4230
|
return 1
|
|
4139
4231
|
}
|
|
4140
4232
|
|
|
4233
|
+
# Live-eval entry guard (macOS DEBT). Refuse artifact eval on darwin when no real
|
|
4234
|
+
# timeout/gtimeout can bound a stuck claude call. Replay (Phase 7 sets replay_mode=1)
|
|
4235
|
+
# never hits this. --report returns before the call site. Do not move this into
|
|
4236
|
+
# _eval_engine_run or _run_with_timeout.
|
|
4237
|
+
_eval_guard_unbounded_darwin() { # $1=replay_mode (0/1); return 1 = refused
|
|
4238
|
+
local replay_mode="${1:-0}"
|
|
4239
|
+
[ "$replay_mode" -eq 0 ] || return 0
|
|
4240
|
+
[ "${FLOW_EVAL_UNBOUNDED:-}" = "1" ] && return 0
|
|
4241
|
+
local darwin=0
|
|
4242
|
+
if [ "${FLOW_EVAL_FORCE_DARWIN:-}" = "1" ]; then
|
|
4243
|
+
darwin=1
|
|
4244
|
+
else
|
|
4245
|
+
case "$(uname -s 2>/dev/null)" in Darwin) darwin=1 ;; esac
|
|
4246
|
+
fi
|
|
4247
|
+
[ "$darwin" -eq 1 ] || return 0
|
|
4248
|
+
if command -v timeout >/dev/null 2>&1 || command -v gtimeout >/dev/null 2>&1; then
|
|
4249
|
+
return 0
|
|
4250
|
+
fi
|
|
4251
|
+
echo "REFUSED: live eval on macOS without timeout/gtimeout cannot bound a stuck claude call."
|
|
4252
|
+
echo " Unbounded-billing risk: a hung judge can run past --timeout (DEBT.md)."
|
|
4253
|
+
echo " Install GNU coreutils (gtimeout) or export FLOW_EVAL_UNBOUNDED=1 to accept that risk."
|
|
4254
|
+
return 1
|
|
4255
|
+
}
|
|
4256
|
+
|
|
4141
4257
|
# NOTE: defined with the `function name { }` form (no parens) rather than this file's usual
|
|
4142
4258
|
# `name() { }` style, solely so the literal 4-byte substring the shipped verb name is built
|
|
4143
4259
|
# from never appears immediately followed by '(' in the source - a blind text-pattern security
|
|
@@ -4145,7 +4261,7 @@ function cmd_eval_converge { # same arg shape as cmd_eval, called only when --st
|
|
|
4145
4261
|
# -positives on that exact byte sequence regardless of language. Purely a spelling dodge for a
|
|
4146
4262
|
# generic scanner; behavior is 100% identical to every other function in this file.
|
|
4147
4263
|
function cmd_eval {
|
|
4148
|
-
local stage_filter="" fixture_filter="" n=3 timeout=120 report_mode=0 keep_going=0
|
|
4264
|
+
local stage_filter="" fixture_filter="" n=3 timeout=120 report_mode=0 keep_going=0 replay_mode=0 record_mode=0
|
|
4149
4265
|
while [ $# -gt 0 ]; do
|
|
4150
4266
|
case "$1" in
|
|
4151
4267
|
--stage) shift; stage_filter="${1:-}" ;;
|
|
@@ -4153,8 +4269,10 @@ function cmd_eval {
|
|
|
4153
4269
|
--n) shift; n="${1:-3}" ;;
|
|
4154
4270
|
--timeout) shift; timeout="${1:-120}" ;;
|
|
4155
4271
|
--report) report_mode=1 ;;
|
|
4272
|
+
--record) record_mode=1 ;;
|
|
4273
|
+
--replay) replay_mode=1 ;;
|
|
4156
4274
|
--keep-going) keep_going=1 ;;
|
|
4157
|
-
*) echo "usage: /flow eval [--stage 01|02|05|card|routing|converge] [--fixture <id>] [--n 3] [--timeout <seconds>] [--keep-going] [--report]"; return 1 ;;
|
|
4275
|
+
*) echo "usage: /flow eval [--stage 01|02|05|card|routing|converge] [--fixture <id>] [--n 3] [--timeout <seconds>] [--keep-going] [--report] [--record|--replay]"; return 1 ;;
|
|
4158
4276
|
esac
|
|
4159
4277
|
shift 2>/dev/null || true
|
|
4160
4278
|
done
|
|
@@ -4167,6 +4285,26 @@ function cmd_eval {
|
|
|
4167
4285
|
local backoff="${FLOW_EVAL_RETRY_BACKOFF:-5}"
|
|
4168
4286
|
case "$backoff" in ''|*[!0-9]*) backoff=5 ;; esac
|
|
4169
4287
|
|
|
4288
|
+
# Reject window: --record|--replay is artifact-modality only. Must run BEFORE routing/
|
|
4289
|
+
# converge dispatch and BEFORE the --report early-return. --record is LIVE (replay_mode
|
|
4290
|
+
# stays 0) so the Phase 6 darwin guard still fires.
|
|
4291
|
+
if [ "$record_mode" -eq 1 ] && [ "$replay_mode" -eq 1 ]; then
|
|
4292
|
+
echo "usage: /flow eval --record and --replay are mutually exclusive"
|
|
4293
|
+
return 1
|
|
4294
|
+
fi
|
|
4295
|
+
if [ "$record_mode" -eq 1 ] || [ "$replay_mode" -eq 1 ]; then
|
|
4296
|
+
case "$stage_filter" in
|
|
4297
|
+
routing|converge)
|
|
4298
|
+
echo "usage: /flow eval --record|--replay is artifact-modality only (not --stage $stage_filter)"
|
|
4299
|
+
return 1
|
|
4300
|
+
;;
|
|
4301
|
+
esac
|
|
4302
|
+
if [ "$report_mode" -eq 1 ]; then
|
|
4303
|
+
echo "usage: /flow eval --record|--replay cannot combine with --report"
|
|
4304
|
+
return 1
|
|
4305
|
+
fi
|
|
4306
|
+
fi
|
|
4307
|
+
|
|
4170
4308
|
# v0.22: routing is a SEPARATE judge modality (own manifest/prompt/verdict/results-file/
|
|
4171
4309
|
# scorecard) - delegate entirely and return, before any of the artifact-eval logic below
|
|
4172
4310
|
# (which assumes gate-rules.md stage sections and FLAG/PASS verdicts) ever runs.
|
|
@@ -4200,22 +4338,56 @@ function cmd_eval {
|
|
|
4200
4338
|
return 1
|
|
4201
4339
|
fi
|
|
4202
4340
|
|
|
4203
|
-
|
|
4204
|
-
|
|
4205
|
-
|
|
4206
|
-
|
|
4207
|
-
|
|
4208
|
-
fail)
|
|
4209
|
-
echo "SKIP: 'claude' CLI is present but the sentinel probe did not come back clean (one minimal"
|
|
4210
|
-
echo " billable probe call was made). Confirm 'claude -p' runs headless on this machine, then retry."
|
|
4211
|
-
return 0 ;;
|
|
4212
|
-
esac
|
|
4341
|
+
# Live-eval entry. replay_mode=1 self-skips inside the guard; --record leaves
|
|
4342
|
+
# replay_mode=0 so refuse-by-default still fires. Do not wrap this call again.
|
|
4343
|
+
if ! _eval_guard_unbounded_darwin "$replay_mode"; then
|
|
4344
|
+
return 1
|
|
4345
|
+
fi
|
|
4213
4346
|
|
|
4214
|
-
local nonce
|
|
4215
|
-
local total_mismatch=0 total_unreliable=0 total_evaluated=0 rundir="" n_written=0 aborted=0
|
|
4347
|
+
local nonce="" total_mismatch=0 total_unreliable=0 total_evaluated=0 rundir="" n_written=0 aborted=0
|
|
4216
4348
|
# Computed ONCE per batch, not per fixture row - see _eval_emit_result's comment.
|
|
4217
|
-
local batch_cliv batch_flowv batch_grsha
|
|
4218
|
-
|
|
4349
|
+
local batch_cliv="" batch_flowv="" batch_grsha="" prompt_sha_acc="" prompt_sha_enforced=0
|
|
4350
|
+
if [ "$replay_mode" -eq 1 ]; then
|
|
4351
|
+
# Replay wrap: skip probe SKIP-exit and _eval_cli_version only. Still compute
|
|
4352
|
+
# batch_grsha (staleness hard-fail) and _flow_version. Nonce from fixture, never _eval_nonce.
|
|
4353
|
+
if [ ! -f "$EVAL_REPLAY_DIR/meta" ]; then
|
|
4354
|
+
echo "eval --replay: missing replay fixtures at $EVAL_REPLAY_DIR (no meta). Never SKIP."
|
|
4355
|
+
return 1
|
|
4356
|
+
fi
|
|
4357
|
+
nonce="$(_eval_replay_meta_get nonce)" || nonce=""
|
|
4358
|
+
if [ -z "$nonce" ]; then
|
|
4359
|
+
echo "eval --replay: meta missing nonce. Never SKIP."
|
|
4360
|
+
return 1
|
|
4361
|
+
fi
|
|
4362
|
+
batch_grsha="$(_eval_gate_rules_sha)"
|
|
4363
|
+
batch_flowv="$(_flow_version)"
|
|
4364
|
+
batch_cliv="replay"
|
|
4365
|
+
local rec_sha
|
|
4366
|
+
rec_sha="$(_eval_replay_meta_get gate_rules_sha)" || rec_sha=""
|
|
4367
|
+
if [ "$rec_sha" != "$batch_grsha" ]; then
|
|
4368
|
+
echo "eval --replay: fixtures stale — gate-rules changed; re-record live per ADR re-baseline rule"
|
|
4369
|
+
echo " recorded=$rec_sha current=$batch_grsha"
|
|
4370
|
+
return 1
|
|
4371
|
+
fi
|
|
4372
|
+
if grep -q '^prompt_sha\.' "$EVAL_REPLAY_DIR/meta" 2>/dev/null; then
|
|
4373
|
+
prompt_sha_enforced=1
|
|
4374
|
+
fi
|
|
4375
|
+
echo "replay: keyless artifact batch (nonce from fixture, zero live calls)"
|
|
4376
|
+
else
|
|
4377
|
+
_eval_isolated_guard || return 1
|
|
4378
|
+
local probe; probe="$(_eval_isolated_run _eval_probe)"
|
|
4379
|
+
case "$probe" in
|
|
4380
|
+
absent)
|
|
4381
|
+
echo "SKIP: 'claude' CLI not found on PATH - eval needs it to run the LLM judge. Zero calls made."
|
|
4382
|
+
return 0 ;;
|
|
4383
|
+
fail)
|
|
4384
|
+
echo "SKIP: 'claude' CLI is present but the sentinel probe did not come back clean (one minimal"
|
|
4385
|
+
echo " billable probe call was made). Confirm 'claude -p' runs headless on this machine, then retry."
|
|
4386
|
+
return 0 ;;
|
|
4387
|
+
esac
|
|
4388
|
+
nonce="$(_eval_nonce)"
|
|
4389
|
+
batch_cliv="$(_eval_cli_version)"; batch_flowv="$(_flow_version)"; batch_grsha="$(_eval_gate_rules_sha)"
|
|
4390
|
+
fi
|
|
4219
4391
|
echo "eval: batch $nonce (N=$n per fixture, timeout=${timeout}s)"
|
|
4220
4392
|
|
|
4221
4393
|
# Ensure .flow/ (which now hosts eval-raw/ postmortem dumps) is git-ignored on any project
|
|
@@ -4246,7 +4418,9 @@ function cmd_eval {
|
|
|
4246
4418
|
}
|
|
4247
4419
|
END { print c + 0 }
|
|
4248
4420
|
' "$EVAL_MANIFEST")"
|
|
4249
|
-
|
|
4421
|
+
if [ "$replay_mode" -eq 0 ]; then
|
|
4422
|
+
_eval_emit_batch_marker "$nonce" start "$n_expected"
|
|
4423
|
+
fi
|
|
4250
4424
|
|
|
4251
4425
|
# Scoped INT/TERM trap: the sole EXIT trap (_log_on_exit -> _cleanup_tds) was found NOT to
|
|
4252
4426
|
# fire reliably while blocked on a foreground `_run_with_timeout` child (verified: a signal
|
|
@@ -4293,29 +4467,63 @@ function cmd_eval {
|
|
|
4293
4467
|
|
|
4294
4468
|
local promptfile="$rundir/prompt.txt"
|
|
4295
4469
|
if ! _eval_build_prompt "$promptfile" "$fstage_short" "$artifact_path" "$nonce"; then
|
|
4470
|
+
if [ "$replay_mode" -eq 1 ]; then
|
|
4471
|
+
echo " $fid: SKIP - no heading-mapped section (artifact-replay scope is 01|02|card)"
|
|
4472
|
+
rm -rf "$rundir" 2>/dev/null
|
|
4473
|
+
continue
|
|
4474
|
+
fi
|
|
4296
4475
|
echo " $fid: FAIL - gate-rules.md section for stage '$fstage' extracted EMPTY (heading map or file drift) - aborted before any billable call"
|
|
4297
4476
|
total_mismatch=$((total_mismatch + 1))
|
|
4298
4477
|
rm -rf "$rundir" 2>/dev/null
|
|
4299
4478
|
continue
|
|
4300
4479
|
fi
|
|
4301
4480
|
|
|
4481
|
+
local psha
|
|
4482
|
+
psha="$(_eval_prompt_sha "$promptfile")"
|
|
4483
|
+
if [ "$replay_mode" -eq 1 ] && [ "$prompt_sha_enforced" -eq 1 ]; then
|
|
4484
|
+
local rec_psha
|
|
4485
|
+
rec_psha="$(_eval_replay_meta_get "prompt_sha.$fid")" || rec_psha=""
|
|
4486
|
+
if [ -z "$rec_psha" ] || [ "$rec_psha" != "$psha" ]; then
|
|
4487
|
+
echo "eval --replay: fixtures stale — prompt assembly changed; re-record live per ADR re-baseline rule"
|
|
4488
|
+
echo " recorded=$rec_psha current=$psha fixture=$fid"
|
|
4489
|
+
rm -rf "$rundir" 2>/dev/null
|
|
4490
|
+
trap - INT TERM
|
|
4491
|
+
return 1
|
|
4492
|
+
fi
|
|
4493
|
+
fi
|
|
4494
|
+
if [ "$record_mode" -eq 1 ]; then
|
|
4495
|
+
prompt_sha_acc="${prompt_sha_acc}prompt_sha.${fid}=${psha}"$'\n'
|
|
4496
|
+
fi
|
|
4497
|
+
|
|
4302
4498
|
total_evaluated=$((total_evaluated + 1))
|
|
4303
|
-
local flag_count=0 pass_count=0 invalid_count=0 i=1 model="unknown" retries_sum=0 vote_rate_limited=false
|
|
4499
|
+
local flag_count=0 pass_count=0 invalid_count=0 i=1 model="unknown" retries_sum=0 vote_rate_limited=false vote_timed_out=false
|
|
4304
4500
|
# Filename-safe fixture id for raw-capture paths - manifest fid is only tr -d '\r'-cleaned
|
|
4305
4501
|
# (the v1 trust-boundary is READ-side only, but we are now WRITING keyed by it), so a
|
|
4306
4502
|
# hand-edited or FLOW_EVAL_MANIFEST-overridden manifest cannot traverse out of eval-raw/.
|
|
4307
4503
|
local fid_safe; fid_safe="$(printf '%s' "$fid" | tr -c 'A-Za-z0-9' '-' | sed -E 's/-+/-/g; s/^-//; s/-$//')"
|
|
4308
4504
|
[ -z "$fid_safe" ] && fid_safe="_"
|
|
4505
|
+
if [ "$replay_mode" -eq 0 ]; then
|
|
4506
|
+
local rawdir_prompt="$LOG_DIR/eval-raw/$nonce"
|
|
4507
|
+
if mkdir -p "$rawdir_prompt" 2>/dev/null; then
|
|
4508
|
+
cp "$promptfile" "$rawdir_prompt/${fid_safe}.prompt" 2>/dev/null || true
|
|
4509
|
+
fi
|
|
4510
|
+
fi
|
|
4309
4511
|
while [ "$i" -le "$n" ]; do
|
|
4310
4512
|
local raw rc v raw1="" rc1=0 err1="" err2=""
|
|
4311
4513
|
# Attempt 1 - stderr captured at the OUTER redirection so it never enters the sh -c
|
|
4312
4514
|
# command string _eval_engine_run passes to _run_with_timeout (that broke the watchdog
|
|
4313
4515
|
# fallback on the timeout-less-PATH lane - see engine-run's note).
|
|
4314
4516
|
err1="$rundir/v${i}-a1.err"
|
|
4315
|
-
|
|
4517
|
+
if [ "$replay_mode" -eq 1 ]; then
|
|
4518
|
+
raw="$(_eval_replay_vote "$fid" "$i")" || { rm -rf "$rundir" 2>/dev/null; trap - INT TERM; return 1; }
|
|
4519
|
+
rc=0
|
|
4520
|
+
else
|
|
4521
|
+
raw="$(_eval_isolated_run _eval_engine_run "$promptfile" "$timeout" 2>"$err1")"; rc=$?
|
|
4522
|
+
fi
|
|
4316
4523
|
raw1="$raw"; rc1="$rc"
|
|
4317
4524
|
if [ "$rc" -eq 124 ]; then
|
|
4318
4525
|
v=INVALID
|
|
4526
|
+
vote_timed_out=true
|
|
4319
4527
|
else
|
|
4320
4528
|
v="$(_eval_parse_verdict "$raw" "$nonce")"
|
|
4321
4529
|
[ "$model" = "unknown" ] && model="$(_eval_parse_model "$raw")"
|
|
@@ -4332,7 +4540,7 @@ function cmd_eval {
|
|
|
4332
4540
|
# _run_with_timeout is documented not to bound a stuck call (DEBT.md), a retry here just
|
|
4333
4541
|
# waits another full stuck-call duration - test E on the 3-OS CI matrix reproduces this at
|
|
4334
4542
|
# ~66s (2x ~30s stuck calls + backoff) despite --timeout 3.
|
|
4335
|
-
if [ "$v" = "INVALID" ] && [ "$rl1" != "true" ] && [ "$rc1" -ne 124 ]; then
|
|
4543
|
+
if [ "$replay_mode" -eq 0 ] && [ "$v" = "INVALID" ] && [ "$rl1" != "true" ] && [ "$rc1" -ne 124 ]; then
|
|
4336
4544
|
# Backoff then retry - env-injectable so tests can zero it; skipped when rc=124 (timeout
|
|
4337
4545
|
# was infra, retry would timeout again) or when rate-limited (see above).
|
|
4338
4546
|
if [ "$backoff" -gt 0 ]; then
|
|
@@ -4342,9 +4550,14 @@ function cmd_eval {
|
|
|
4342
4550
|
echo " $fid: retrying vote $i (parse-INVALID on attempt 1)"
|
|
4343
4551
|
fi
|
|
4344
4552
|
err2="$rundir/v${i}-a2.err"
|
|
4345
|
-
|
|
4553
|
+
if [ "$replay_mode" -eq 1 ]; then
|
|
4554
|
+
raw="$(_eval_replay_vote "$fid" "$i")" || { rm -rf "$rundir" 2>/dev/null; trap - INT TERM; return 1; }
|
|
4555
|
+
rc=0
|
|
4556
|
+
else
|
|
4557
|
+
raw="$(_eval_isolated_run _eval_engine_run "$promptfile" "$timeout" 2>"$err2")"; rc=$?
|
|
4558
|
+
fi
|
|
4346
4559
|
retries_sum=$((retries_sum + 1))
|
|
4347
|
-
if [ "$rc" -eq 124 ]; then v=INVALID; else v="$(_eval_parse_verdict "$raw" "$nonce")"; fi
|
|
4560
|
+
if [ "$rc" -eq 124 ]; then v=INVALID; vote_timed_out=true; else v="$(_eval_parse_verdict "$raw" "$nonce")"; fi
|
|
4348
4561
|
[ "$model" = "unknown" ] && [ "$rc" -ne 124 ] && model="$(_eval_parse_model "$raw")"
|
|
4349
4562
|
local rl2; rl2="$(_eval_parse_rate_limited "$raw")"
|
|
4350
4563
|
[ "$rl2" = "true" ] && vote_rate_limited=true
|
|
@@ -4385,6 +4598,9 @@ function cmd_eval {
|
|
|
4385
4598
|
PASS) pass_count=$((pass_count + 1)) ;;
|
|
4386
4599
|
*) invalid_count=$((invalid_count + 1)) ;;
|
|
4387
4600
|
esac
|
|
4601
|
+
if [ "$record_mode" -eq 1 ]; then
|
|
4602
|
+
_eval_record_vote "$fid" "$i" "$nonce" "$v"
|
|
4603
|
+
fi
|
|
4388
4604
|
i=$((i + 1))
|
|
4389
4605
|
done
|
|
4390
4606
|
rm -rf "$rundir" 2>/dev/null
|
|
@@ -4415,7 +4631,9 @@ function cmd_eval {
|
|
|
4415
4631
|
echo " $fid ($fstage): $verdict - MISMATCH, expected $fexpected (flag=$flag_count pass=$pass_count invalid=$invalid_count)"
|
|
4416
4632
|
fi
|
|
4417
4633
|
|
|
4418
|
-
|
|
4634
|
+
if [ "$replay_mode" -eq 0 ]; then
|
|
4635
|
+
_eval_emit_result "$nonce" "$fid" "$fstage" "$fexpected" "$verdict" "$match" "$flag_count" "$pass_count" "$invalid_count" "$n" "$model" "$batch_cliv" "$batch_flowv" "$batch_grsha" "$retries_sum" "$vote_rate_limited" "$psha" "$vote_timed_out"
|
|
4636
|
+
fi
|
|
4419
4637
|
n_written=$((n_written + 1))
|
|
4420
4638
|
|
|
4421
4639
|
# Circuit breaker: catches the 260710-class INVALID storm early. Trip when the FIRST
|
|
@@ -4442,9 +4660,12 @@ function cmd_eval {
|
|
|
4442
4660
|
# SKIPS the trailer so an early-broken batch never satisfies --report's completeness check
|
|
4443
4661
|
# (n_written>=n_expected AND trailer present), even on a --fixture-filtered run where
|
|
4444
4662
|
# n_written could accidentally equal the filtered n_expected of 1.
|
|
4445
|
-
if [ "$aborted" -eq 0 ]; then
|
|
4663
|
+
if [ "$aborted" -eq 0 ] && [ "$replay_mode" -eq 0 ]; then
|
|
4446
4664
|
_eval_emit_batch_marker "$nonce" done "$n_written"
|
|
4447
4665
|
fi
|
|
4666
|
+
if [ "$record_mode" -eq 1 ] && [ "$aborted" -eq 0 ]; then
|
|
4667
|
+
_eval_replay_write_meta "$nonce" "$batch_grsha" "${model:-unknown}" "$n" "$prompt_sha_acc"
|
|
4668
|
+
fi
|
|
4448
4669
|
|
|
4449
4670
|
echo
|
|
4450
4671
|
if [ "$total_evaluated" -eq 0 ]; then
|
|
@@ -4457,8 +4678,13 @@ function cmd_eval {
|
|
|
4457
4678
|
echo "ABORTED: circuit breaker tripped on first fixture UNRELIABLE - $total_evaluated of $n_expected fixtures evaluated."
|
|
4458
4679
|
return 2
|
|
4459
4680
|
fi
|
|
4460
|
-
|
|
4681
|
+
if [ "$replay_mode" -eq 0 ]; then
|
|
4682
|
+
_eval_print_scorecard "$nonce"
|
|
4683
|
+
fi
|
|
4461
4684
|
echo
|
|
4685
|
+
if [ "$replay_mode" -eq 1 ]; then
|
|
4686
|
+
echo "replay: $total_evaluated fixture(s) (keyless)"
|
|
4687
|
+
fi
|
|
4462
4688
|
if [ "$total_mismatch" -eq 0 ] && [ "$total_unreliable" -eq 0 ]; then
|
|
4463
4689
|
echo "PASS: all $total_evaluated evaluated fixture(s) majority-matched their expected verdict."
|
|
4464
4690
|
return 0
|
|
@@ -4506,9 +4732,14 @@ usage: bash flow.sh <command> [args]
|
|
|
4506
4732
|
promote <file> Copy a playbook into the cross-project KB (~/.claude/flow/playbooks)
|
|
4507
4733
|
doctor Check the environment (bash/python/grep/git) across macOS/Linux/Windows
|
|
4508
4734
|
eval [opts] Behavioral eval: does the LLM semantic gate flag a hollow-but-mechanically-clean
|
|
4509
|
-
fixture? Opt-in, BILLABLE (skips cleanly with zero calls if 'claude'
|
|
4735
|
+
fixture? Opt-in, BILLABLE (live mode skips cleanly with zero calls if 'claude'
|
|
4736
|
+
CLI absent; --replay never inherits that SKIP).
|
|
4510
4737
|
[--stage 01|02|05|card|routing|converge] [--fixture <id>] [--n 3] [--timeout <seconds>]
|
|
4511
4738
|
[--report] offline: last complete batch's scorecard + drift, zero calls
|
|
4739
|
+
[--record] live: write stripped transcripts under eval/replay/ (billable)
|
|
4740
|
+
[--replay] keyless: feed recorded transcripts through parse/vote/scorecard
|
|
4741
|
+
--record|--replay are artifact-modality only (not --stage routing|converge,
|
|
4742
|
+
not --report). --replay is not a fresh-judge and never counts toward the floor.
|
|
4512
4743
|
--stage routing: SEPARATE judge (concierge routing, not gate-rules FLAG/PASS);
|
|
4513
4744
|
hard-capped at 90 calls/batch, own results stream, own --report
|
|
4514
4745
|
retro Print the 3 retro questions
|
package/skills-manifest.json
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
{
|
|
2
2
|
"source": "../skills/flow",
|
|
3
3
|
"destRelative": "skills/flow",
|
|
4
|
-
"fileCount":
|
|
5
|
-
"syncedAt": "2026-08-
|
|
4
|
+
"fileCount": 103,
|
|
5
|
+
"syncedAt": "2026-08-20T13:35:30.768Z",
|
|
6
6
|
"fileList": [
|
|
7
7
|
"SKILL.md",
|
|
8
8
|
"_templates/00-idea.md",
|
|
@@ -35,6 +35,8 @@
|
|
|
35
35
|
"eval/fixtures/fcdb/cards/C-001.md",
|
|
36
36
|
"eval/fixtures/fcdc/.flow/.gitkeep",
|
|
37
37
|
"eval/fixtures/fcdc/cards/C-001.md",
|
|
38
|
+
"eval/fixtures/fcdd/cards/C-001.md",
|
|
39
|
+
"eval/fixtures/fcde/cards/C-001.md",
|
|
38
40
|
"eval/fixtures/routing/manifest.tsv",
|
|
39
41
|
"eval/fixtures/routing/states/state-any-midproject.txt",
|
|
40
42
|
"eval/fixtures/routing/states/state-cards-todo.txt",
|
|
@@ -95,6 +97,7 @@
|
|
|
95
97
|
"references/gate-eval.md",
|
|
96
98
|
"references/gate-rules.md",
|
|
97
99
|
"references/ground-truth-gates.md",
|
|
100
|
+
"references/host-agnostic-parallel.md",
|
|
98
101
|
"references/loop-harness-2026-principles.md",
|
|
99
102
|
"references/mode-work.md",
|
|
100
103
|
"references/native-rituals.md",
|