@manhquy/flow-skill 0.7.0 → 0.7.1-next.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +10 -0
- package/package.json +1 -1
- package/skills/flow/SKILL.md +3 -1
- package/skills/flow/references/agent-detection.md +1 -0
- package/skills/flow/references/auto-run.md +1 -0
- package/skills/flow/references/gate-eval.md +48 -0
- package/skills/flow/references/host-agnostic-parallel.md +54 -0
- package/skills/flow/runner/flow.sh +111 -34
- package/skills-manifest.json +3 -2
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,16 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to `@manhquy/flow-skill`. Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Versioning: [SemVer](https://semver.org/).
|
|
4
4
|
|
|
5
|
+
## [0.7.1-next.0] — 2026-08-20 — ships skill v0.31.0 (eval receipts + host-agnostic parallel + value-first docs)
|
|
6
|
+
|
|
7
|
+
Skill-content release. Re-ships skill product **v0.31.0** (see root
|
|
8
|
+
`CHANGELOG.md` `## 0.31.0`). Installer CLI contract unchanged. Publishes to
|
|
9
|
+
dist-tag **`next`**.
|
|
10
|
+
|
|
11
|
+
### Changed
|
|
12
|
+
- Bundled skill tree re-synced from monorepo `skills/flow/`.
|
|
13
|
+
- Install (prerelease): **`npx @manhquy/flow-skill@next`**.
|
|
14
|
+
|
|
5
15
|
## [0.7.0] — 2026-08-14 — ships skill v0.30.0 (discipline-layer identity + CI/eval hardening)
|
|
6
16
|
|
|
7
17
|
Skill-content release. Ships skill product **v0.30.0**: identity ADR (gates and
|
package/package.json
CHANGED
package/skills/flow/SKILL.md
CHANGED
|
@@ -8,7 +8,7 @@ keywords: [flow, buildflow, gate, build, ship, scope, prd, contract, card, deplo
|
|
|
8
8
|
license: MIT
|
|
9
9
|
metadata:
|
|
10
10
|
author: flow-skill
|
|
11
|
-
version: "0.
|
|
11
|
+
version: "0.31.0"
|
|
12
12
|
attribution: "Methodology from ai20k-build-phase/buildflow (Tony, arealisticdreamer.com); durable-layer ancestry and improve-ritual spirit from repository-harness (protocol v1 EOL — flow-owned fork); agent patterns from claudekit-engineer, BMAD-METHOD. v0.22 concierge from BMAD bmad-help; forge-idea from bmad-forge-idea (MIT, BMad Code LLC)."
|
|
13
13
|
---
|
|
14
14
|
|
|
@@ -190,6 +190,8 @@ law/contract excerpts (no session history); each returns DONE/DONE_WITH_CONCERNS
|
|
|
190
190
|
NEEDS_CONTEXT. After a delegation: run the gate, apply the semantic challenge, write the
|
|
191
191
|
durable hook (`flow.sh harness ...`), announce which path ran.
|
|
192
192
|
|
|
193
|
+
Parallel cards ride the current host (in-process Task, extra terminals, or an already-inside mux); flow requires no named multiplexer — `references/host-agnostic-parallel.md`.
|
|
194
|
+
|
|
193
195
|
Mode `work` (`references/mode-work.md`): interview once → draft 00-05 → one scope pause →
|
|
194
196
|
deliver the card set as one summary. Gates bind the same as `teach`.
|
|
195
197
|
|
|
@@ -74,6 +74,7 @@ Manager / real `agy` terminal, paste the ReviewResult back); an empty Gemini res
|
|
|
74
74
|
- **Same gate regardless of path.** The buildflow gate (`flow.sh` + `gate-rules.md`) is the
|
|
75
75
|
contract. An agent fills an artifact; the gate still judges it. A missing agent never
|
|
76
76
|
lowers a gate — it only changes who drafts.
|
|
77
|
+
- **No named mux required.** Absence of any multiplexer never fails a gate. Parallel occupancy is host-owned — `host-agnostic-parallel.md`.
|
|
77
78
|
- **Context isolation (orchestration-protocol).** Give each subagent ONLY: the task, the
|
|
78
79
|
specific files to read/modify, acceptance criteria, and relevant law/contract excerpts.
|
|
79
80
|
Never the full session history. One card = one scoped brief.
|
|
@@ -101,6 +101,7 @@ and log the durable metric (`flow.sh harness intervention add`) per `codex-integ
|
|
|
101
101
|
parallel, each in its own worktree/session. Merge back in card-number order, one at a time,
|
|
102
102
|
running the merged app once between merges. A merge conflict means the overlap check was
|
|
103
103
|
gamed — stop and re-plan. Cards needing the deployed app (contract-tests, e2e) are serial.
|
|
104
|
+
Placement and wait stay with the host; see `host-agnostic-parallel.md`.
|
|
104
105
|
|
|
105
106
|
**Human-driven `/flow workspace` coexists with this auto loop.** `auto` provisions a worktree per
|
|
106
107
|
card internally (`git worktree add ../<project>-C-NNN -b card/C-NNN`); `/flow workspace add card/C-NNN`
|
|
@@ -90,6 +90,54 @@ unbounded-billing risk. Install GNU coreutils (`gtimeout`) or export `FLOW_EVAL_
|
|
|
90
90
|
accept that risk for one run. `--report` and `--replay` never hit this guard. `--record` is live
|
|
91
91
|
and does.
|
|
92
92
|
|
|
93
|
+
### Cheap isolated live eval (host recipe)
|
|
94
|
+
|
|
95
|
+
Live cost is dominated by **host** context (project `CLAUDE.md`/`AGENTS.md` walk + user
|
|
96
|
+
plugins/skills/MCP), not the inlined judge prompt. Isolation is cwd + env around the
|
|
97
|
+
existing one-shot — never extra engine argv, never `--bare`, never `--model`.
|
|
98
|
+
|
|
99
|
+
`FLOW_EVAL_ISOLATED_CWD` unset → today's cwd (byte-identical). Set to an existing directory
|
|
100
|
+
→ probe and engine run in `( cd "$FLOW_EVAL_ISOLATED_CWD" && export CLAUDE_CODE_SAFE_MODE=1 &&
|
|
101
|
+
… )`. Set but not a directory → fail loud. `CLAUDE_CODE_SAFE_MODE=1` is exported only in
|
|
102
|
+
that subshell (official Claude env; keeps OAuth). Do **not** empty `HOME` or point
|
|
103
|
+
`CLAUDE_CONFIG_DIR` at the scratch dir (Linux credentials live there).
|
|
104
|
+
|
|
105
|
+
```bash
|
|
106
|
+
# Auth must be live (`claude auth login` if `claude auth status` says not logged in).
|
|
107
|
+
ISOLATED="$(mktemp -d /tmp/flow-eval-XXXXXX)" # NOT an ancestor of the flow-skill tree
|
|
108
|
+
export FLOW_EVAL_ISOLATED_CWD="$ISOLATED"
|
|
109
|
+
export FLOW_PROJECT_ROOT="/path/to/real/project" # results stay in $ROOT/.flow/
|
|
110
|
+
bash /path/to/skills/flow/runner/flow.sh eval --n 1 --fixture fcda
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
The first isolated batch is a **new baseline**, not comparable to the 2026-07-10
|
|
114
|
+
loaded-context floor ($0.30–0.37/call, ~50–60K cache-creation). A judge-model change is a
|
|
115
|
+
full re-baseline. Do not mix isolated and loaded-context batches in tripwire-2. Update the
|
|
116
|
+
measured dollars in this Cost section **after** the operator checkpoint publishes isolated
|
|
117
|
+
`total_cost_usd` + `cache_creation_input_tokens`. Replay still never counts.
|
|
118
|
+
|
|
119
|
+
Cost measurement against one captured `--output-format json` document (do not commit the
|
|
120
|
+
envelope — `session_id`/`cwd` leak):
|
|
121
|
+
|
|
122
|
+
```bash
|
|
123
|
+
printf '%s' "$raw" | python3 -c '
|
|
124
|
+
import json,sys
|
|
125
|
+
d=json.load(sys.stdin)
|
|
126
|
+
if isinstance(d,list):
|
|
127
|
+
d=next(x for x in d if isinstance(x,dict) and x.get("type")=="result")
|
|
128
|
+
u=d.get("usage") or {}
|
|
129
|
+
print("usd", d.get("total_cost_usd"))
|
|
130
|
+
print("in", u.get("input_tokens"), "out", u.get("output_tokens"))
|
|
131
|
+
print("cache_create", u.get("cache_creation_input_tokens") or u.get("cache_creation"))
|
|
132
|
+
print("cache_read", u.get("cache_read_input_tokens"))
|
|
133
|
+
print("model", d.get("model"))
|
|
134
|
+
'
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
Success signal: `cache_creation_input_tokens` collapses from ~50–60K toward the inlined
|
|
138
|
+
judge prompt size; `total_cost_usd` drops well below $0.30.
|
|
139
|
+
|
|
140
|
+
|
|
93
141
|
## Modes
|
|
94
142
|
|
|
95
143
|
| Mode | Calls | What it proves |
|
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
# Host-agnostic parallel occupancy
|
|
2
|
+
|
|
3
|
+
Flow requires **no named multiplexer**. Parallel cards are a ready-gate +
|
|
4
|
+
worktrees + host-owned placement + world-state fan-in. Absence of Herdr, tmux,
|
|
5
|
+
in-process `Task`, or any other host **never fails a gate** and never changes
|
|
6
|
+
`flow.sh` exit.
|
|
7
|
+
|
|
8
|
+
Herdr and tmux appear below only as **examples, never as a dependency** or a
|
|
9
|
+
first-class adapter.
|
|
10
|
+
|
|
11
|
+
## Degrade ladder
|
|
12
|
+
|
|
13
|
+
| Rung | Primitive already in the process | Who binds occupants | Who waits |
|
|
14
|
+
|---|---|---|---|
|
|
15
|
+
| **0** | In-process subagents / `Task` | Hosting agent; cwd = the card's worktree | Host `Task` join |
|
|
16
|
+
| **1** | Operator extra terminals / tabs | Human pastes the print-enter block | Human |
|
|
17
|
+
| **2** | Process is **already inside** a mux (`HERDR_ENV=1` / `TMUX` set) | That mux's **own** skill/CLI; flow does not wrap or detect it | That mux's own wait |
|
|
18
|
+
| **3** | None of the above, or `/flow ready` overlap | Serial: one card | Hosting agent, one at a time |
|
|
19
|
+
|
|
20
|
+
Rung 2 is opt-in by the mux's own environment injection, never by `PATH`.
|
|
21
|
+
Inside-mux probe is the mux's own env (`HERDR_ENV=1`, `TMUX`), never
|
|
22
|
+
`command -v herdr` from outside (a CLI can talk to a same-uid socket without
|
|
23
|
+
`HERDR_ENV`). Absence of rungs 0–2 is the default path, not a degrade footnote.
|
|
24
|
+
|
|
25
|
+
## Gate parity
|
|
26
|
+
|
|
27
|
+
Host `idle` / `done` / `blocked` is **not a card pass**. `flow.sh check` still
|
|
28
|
+
judges, and it must run **in the card's worktree** (parent-tree check of main
|
|
29
|
+
is a false fail/pass). Screen output is not evidence.
|
|
30
|
+
|
|
31
|
+
Keep the four "blocked" senses distinct:
|
|
32
|
+
|
|
33
|
+
| Sense | Meaning | Home |
|
|
34
|
+
|---|---|---|
|
|
35
|
+
| **host-blocked** | mux/pane approval chrome or occupant stuck | the host |
|
|
36
|
+
| **subagent-BLOCKED** | scoped subagent status protocol | `agent-detection.md` |
|
|
37
|
+
| **ready-blocked** | deps unmet or allowed-files overlap | `flow.sh ready` |
|
|
38
|
+
| **Tier-C security halt** | security-class exposure; operator DEBT | `auto-run.md` |
|
|
39
|
+
|
|
40
|
+
Mixing them auto-answers a payments card or auto-halts a yes/no prompt.
|
|
41
|
+
|
|
42
|
+
## File-is-the-wait
|
|
43
|
+
|
|
44
|
+
The **HOST waits** (Task join / human / the mux's own wait). `flow.sh` **never waits on or polls** another agent. Fan-in is world-state: re-invoke `check` in
|
|
45
|
+
the card worktree after the host's wait returns.
|
|
46
|
+
|
|
47
|
+
## Forbidden (ADR-0001; not a nit)
|
|
48
|
+
|
|
49
|
+
`flow.sh` must not exec host control CLIs (`herdr agent start`, `herdr agent wait`, `herdr agent prompt`, `herdr server stop`, `tmux send-keys`, and equivalents).
|
|
50
|
+
Forbidden by name: `mux-up`, `mux-run-wave`, `mux-proxy`. No screen output as
|
|
51
|
+
evidence. No hook install into `~/.claude` or `~/.omp`. No long-lived socket subscribe (`events.subscribe`). `flow.sh` never spawns agents; placement is print-enter or the host's own Task/mux.
|
|
52
|
+
|
|
53
|
+
Own-runtime / own-PTY / own-wait is a successor ADR plus a separate product,
|
|
54
|
+
not a card in this skill.
|
|
@@ -2302,6 +2302,14 @@ cmd_doctor() {
|
|
|
2302
2302
|
command -v cargo >/dev/null 2>&1 && echo " cargo: ok (optional Rust harness power-path)" || true
|
|
2303
2303
|
[ -d "$TEMPLATE_DIR" ] && echo " templates: ok" || { echo " templates: MISSING ($TEMPLATE_DIR)"; ok=0; }
|
|
2304
2304
|
[ -f "$HARNESS_PY" ] && echo " harness: present ($HARNESS_PY)" || echo " harness: absent (durable layer off)"
|
|
2305
|
+
if command -v timeout >/dev/null 2>&1 || command -v gtimeout >/dev/null 2>&1; then
|
|
2306
|
+
echo " timeout: full (timeout|gtimeout present)"
|
|
2307
|
+
else
|
|
2308
|
+
case "$(uname -s 2>/dev/null)" in
|
|
2309
|
+
Darwin) echo " timeout: absent (no timeout/gtimeout; live eval REFUSED)" ;;
|
|
2310
|
+
*) echo " timeout: partial (watchdog fallback; live eval not GNU-bounded)" ;;
|
|
2311
|
+
esac
|
|
2312
|
+
fi
|
|
2305
2313
|
echo
|
|
2306
2314
|
if [ "$ok" -eq 1 ]; then
|
|
2307
2315
|
echo "READY. Gate engine works$([ -n "$py" ] && echo '; durable layer on.' || echo '; durable layer off (no python).')"
|
|
@@ -3199,6 +3207,25 @@ _eval_engine_run() { # $1=promptfile $2=timeout-seconds -> stdout=raw json; retu
|
|
|
3199
3207
|
# command string above. Empirical: adding a `2> '$errfile'` inside the timed cmd breaks the
|
|
3200
3208
|
# watchdog-fallback path in `_run_with_timeout` on the timeout-less-PATH lane (macOS DEBT lane).
|
|
3201
3209
|
# Outer-scope redirection is exactly the same capture at zero cost to the timeout contract.
|
|
3210
|
+
# Opt-in isolated cwd for live eval (Approach D). Unset = today's cwd (byte-identical
|
|
3211
|
+
# call path). Set but not a directory = fail loud. Isolation is cd+env around existing
|
|
3212
|
+
# probe/engine calls — never argv, never a second engine. CLAUDE_CODE_SAFE_MODE=1 is
|
|
3213
|
+
# exported ONLY in the isolated subshell (official Claude env; keeps OAuth; blast radius
|
|
3214
|
+
# is the judge child). Do not invent a flow alias; do not set HOME or CLAUDE_CONFIG_DIR.
|
|
3215
|
+
_eval_isolated_guard() {
|
|
3216
|
+
[ -z "${FLOW_EVAL_ISOLATED_CWD:-}" ] && return 0
|
|
3217
|
+
[ -d "$FLOW_EVAL_ISOLATED_CWD" ] && return 0
|
|
3218
|
+
echo "FAIL: FLOW_EVAL_ISOLATED_CWD is set but is not a directory: $FLOW_EVAL_ISOLATED_CWD"
|
|
3219
|
+
return 1
|
|
3220
|
+
}
|
|
3221
|
+
_eval_isolated_run() {
|
|
3222
|
+
if [ -n "${FLOW_EVAL_ISOLATED_CWD:-}" ]; then
|
|
3223
|
+
( cd "$FLOW_EVAL_ISOLATED_CWD" && export CLAUDE_CODE_SAFE_MODE=1 && "$@" )
|
|
3224
|
+
else
|
|
3225
|
+
"$@"
|
|
3226
|
+
fi
|
|
3227
|
+
}
|
|
3228
|
+
|
|
3202
3229
|
|
|
3203
3230
|
# Build the judge prompt for one fixture. Returns 1 (writes nothing usable) if the gate-rules.md
|
|
3204
3231
|
# section extraction came back empty.
|
|
@@ -3291,6 +3318,13 @@ _eval_gate_rules_sha() {
|
|
|
3291
3318
|
tr -d '\r' < "$GATE_RULES_FILE" | cksum | awk '{print $1}'
|
|
3292
3319
|
}
|
|
3293
3320
|
|
|
3321
|
+
# CRLF-normalized assembler-output hash (same recipe as _eval_gate_rules_sha). Covers
|
|
3322
|
+
# prompt-file bytes only — not host CLAUDE.md. $1=prompt file.
|
|
3323
|
+
_eval_prompt_sha() {
|
|
3324
|
+
[ -f "$1" ] || { printf 'unknown'; return; }
|
|
3325
|
+
tr -d '\r' < "$1" | cksum | awk '{print $1}'
|
|
3326
|
+
}
|
|
3327
|
+
|
|
3294
3328
|
# `claude --version` is the CLI version (e.g. "2.1.201 (Claude Code)"), NOT a model id - kept
|
|
3295
3329
|
# as its own axis distinct from `_eval_parse_model` (red-team finding #4: 3 reviewers
|
|
3296
3330
|
# independently confirmed conflating the two is wrong). `< /dev/null`: this and every other
|
|
@@ -3317,9 +3351,10 @@ _eval_replay_meta_get() { # $1=key -> value (empty + return 1 if missing)
|
|
|
3317
3351
|
return 1
|
|
3318
3352
|
}
|
|
3319
3353
|
|
|
3320
|
-
_eval_replay_write_meta() { # $1=nonce $2=gate_rules_sha $3=model $4=n
|
|
3354
|
+
_eval_replay_write_meta() { # $1=nonce $2=gate_rules_sha $3=model $4=n [$5=prompt_sha.<fid>= lines]
|
|
3321
3355
|
mkdir -p "$EVAL_REPLAY_DIR" || return 1
|
|
3322
|
-
printf 'nonce=%s\ngate_rules_sha=%s\nmodel=%s\nn=%s\n' "$1" "$2" "$3" "$4" > "$EVAL_REPLAY_DIR/meta"
|
|
3356
|
+
printf 'nonce=%s\ngate_rules_sha=%s\nmodel=%s\nn=%s\n' "$1" "$2" "$3" "$4" > "$EVAL_REPLAY_DIR/meta" || return 1
|
|
3357
|
+
[ -n "${5:-}" ] && printf '%s' "$5" >> "$EVAL_REPLAY_DIR/meta"
|
|
3323
3358
|
}
|
|
3324
3359
|
|
|
3325
3360
|
# Persist one stripped verdict line only — never a JSON envelope (session_id/cwd stay out).
|
|
@@ -3366,17 +3401,18 @@ _eval_emit_batch_marker() { # $1=run_id $2=start|done $3=n_expected-or-n_written
|
|
|
3366
3401
|
# cli_version/flow_version/gate_rules_sha are passed in (computed ONCE per batch by the caller,
|
|
3367
3402
|
# not per fixture row) - they never change mid-batch, and recomputing per-row was both wasteful
|
|
3368
3403
|
# (an extra subprocess per fixture) and the original vector for the stdin-consumption bug above.
|
|
3369
|
-
_eval_emit_result() { # $1..$14 as before + $15=retries $16=rate_limited
|
|
3404
|
+
_eval_emit_result() { # $1..$14 as before + $15=retries $16=rate_limited $17=prompt_sha $18=timed_out
|
|
3370
3405
|
local run_id="$1" fid="$2" stage="$3" expected="$4" verdict="$5" match="$6" flag="$7" pass="$8" invalid="$9" n="${10}" model="${11}" cliv="${12}" flowv="${13}" grsha="${14}"
|
|
3371
|
-
local retries="${15:-0}" ratel="${16:-false}"
|
|
3406
|
+
local retries="${15:-0}" ratel="${16:-false}" psha="${17:-}" timedout="${18:-false}"
|
|
3372
3407
|
case "$retries" in ''|*[!0-9]*) retries=0 ;; esac
|
|
3373
3408
|
case "$ratel" in true|false) : ;; *) ratel=false ;; esac
|
|
3409
|
+
case "$timedout" in true|false) : ;; *) timedout=false ;; esac
|
|
3374
3410
|
mkdir -p "$LOG_DIR" 2>/dev/null || true
|
|
3375
|
-
printf '{"ts":"%s","epoch_s":%s,"run_id":"%s","fixture":"%s","stage":"%s","expected":"%s","verdict":"%s","match":"%s","votes":{"flag":%s,"pass":%s,"invalid":%s},"n":%s,"cli_version":"%s","model":"%s","flow_version":"%s","gate_rules_sha":"%s","retries":%s,"rate_limited":%s}\n' \
|
|
3411
|
+
printf '{"ts":"%s","epoch_s":%s,"run_id":"%s","fixture":"%s","stage":"%s","expected":"%s","verdict":"%s","match":"%s","votes":{"flag":%s,"pass":%s,"invalid":%s},"n":%s,"cli_version":"%s","model":"%s","flow_version":"%s","gate_rules_sha":"%s","retries":%s,"rate_limited":%s,"prompt_sha":"%s","timed_out":%s}\n' \
|
|
3376
3412
|
"$(date -u +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || echo unknown)" "$(_now)" "$(_json_str "$run_id")" "$(_json_str "$fid")" \
|
|
3377
3413
|
"$(_json_str "$stage")" "$(_json_str "$expected")" "$(_json_str "$verdict")" "$(_json_str "$match")" \
|
|
3378
3414
|
"$flag" "$pass" "$invalid" "$n" "$(_json_str "$cliv")" "$(_json_str "$model")" \
|
|
3379
|
-
"$(_json_str "$flowv")" "$(_json_str "$grsha")" "$retries" "$ratel" >> "$EVAL_RESULTS_FILE" 2>/dev/null || true
|
|
3415
|
+
"$(_json_str "$flowv")" "$(_json_str "$grsha")" "$retries" "$ratel" "$(_json_str "$psha")" "$timedout" >> "$EVAL_RESULTS_FILE" 2>/dev/null || true
|
|
3380
3416
|
}
|
|
3381
3417
|
|
|
3382
3418
|
# Strip the noisy `system`/`init` envelope from a raw --output-format json line before persisting
|
|
@@ -3660,17 +3696,18 @@ _eval_routing_parse_action() {
|
|
|
3660
3696
|
[ -n "$action" ] && printf '%s' "$action" || echo INVALID
|
|
3661
3697
|
}
|
|
3662
3698
|
|
|
3663
|
-
_eval_routing_emit_result() { # $1=run_id $2=fid $3=expected $4=verdict $5=match $6..$9=votes
|
|
3699
|
+
_eval_routing_emit_result() { # $1=run_id $2=fid $3=expected $4=verdict $5=match $6..$9=votes $10=model $11=cliv $12=flowv $13=rulessha $14=retries $15=rate_limited $16=prompt_sha $17=timed_out
|
|
3664
3700
|
local run_id="$1" fid="$2" expected="$3" verdict="$4" match="$5" hit="$6" miss="$7" invalid="$8" n="$9"
|
|
3665
|
-
local model="${10}" cliv="${11}" flowv="${12}" rulessha="${13}" retries="${14:-0}" ratel="${15:-false}"
|
|
3701
|
+
local model="${10}" cliv="${11}" flowv="${12}" rulessha="${13}" retries="${14:-0}" ratel="${15:-false}" psha="${16:-}" timedout="${17:-false}"
|
|
3666
3702
|
case "$retries" in ''|*[!0-9]*) retries=0 ;; esac
|
|
3667
3703
|
case "$ratel" in true|false) : ;; *) ratel=false ;; esac
|
|
3704
|
+
case "$timedout" in true|false) : ;; *) timedout=false ;; esac
|
|
3668
3705
|
mkdir -p "$LOG_DIR" 2>/dev/null || true
|
|
3669
|
-
printf '{"ts":"%s","epoch_s":%s,"run_id":"%s","fixture":"%s","expected":"%s","verdict":"%s","match":"%s","votes":{"hit":%s,"miss":%s,"invalid":%s},"n":%s,"cli_version":"%s","model":"%s","flow_version":"%s","rules_sha":"%s","retries":%s,"rate_limited":%s}\n' \
|
|
3706
|
+
printf '{"ts":"%s","epoch_s":%s,"run_id":"%s","fixture":"%s","expected":"%s","verdict":"%s","match":"%s","votes":{"hit":%s,"miss":%s,"invalid":%s},"n":%s,"cli_version":"%s","model":"%s","flow_version":"%s","rules_sha":"%s","retries":%s,"rate_limited":%s,"prompt_sha":"%s","timed_out":%s}\n' \
|
|
3670
3707
|
"$(date -u +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || echo unknown)" "$(_now)" "$(_json_str "$run_id")" "$(_json_str "$fid")" \
|
|
3671
3708
|
"$(_json_str "$expected")" "$(_json_str "$verdict")" "$(_json_str "$match")" \
|
|
3672
3709
|
"$hit" "$miss" "$invalid" "$n" "$(_json_str "$cliv")" "$(_json_str "$model")" \
|
|
3673
|
-
"$(_json_str "$flowv")" "$(_json_str "$rulessha")" "$retries" "$ratel" >> "$EVAL_ROUTING_RESULTS_FILE" 2>/dev/null || true
|
|
3710
|
+
"$(_json_str "$flowv")" "$(_json_str "$rulessha")" "$retries" "$ratel" "$(_json_str "$psha")" "$timedout" >> "$EVAL_ROUTING_RESULTS_FILE" 2>/dev/null || true
|
|
3674
3711
|
}
|
|
3675
3712
|
|
|
3676
3713
|
_eval_routing_complete_run_ids() {
|
|
@@ -3770,7 +3807,9 @@ function cmd_eval_routing { # same arg shape as cmd_eval, called only when --sta
|
|
|
3770
3807
|
return 1
|
|
3771
3808
|
fi
|
|
3772
3809
|
|
|
3773
|
-
|
|
3810
|
+
_eval_isolated_guard || return 1
|
|
3811
|
+
|
|
3812
|
+
local probe; probe="$(_eval_isolated_run _eval_probe)"
|
|
3774
3813
|
case "$probe" in
|
|
3775
3814
|
absent)
|
|
3776
3815
|
echo "SKIP: 'claude' CLI not found on PATH - routing eval needs it to run the LLM judge. Zero calls made."
|
|
@@ -3838,14 +3877,18 @@ function cmd_eval_routing { # same arg shape as cmd_eval, called only when --sta
|
|
|
3838
3877
|
continue
|
|
3839
3878
|
fi
|
|
3840
3879
|
|
|
3880
|
+
local psha
|
|
3881
|
+
psha="$(_eval_prompt_sha "$promptfile")"
|
|
3882
|
+
|
|
3841
3883
|
total_evaluated=$((total_evaluated + 1))
|
|
3842
|
-
local hit_count=0 miss_count=0 invalid_count=0 i=1 model="unknown" retries_sum=0 vote_rate_limited=false
|
|
3884
|
+
local hit_count=0 miss_count=0 invalid_count=0 i=1 model="unknown" retries_sum=0 vote_rate_limited=false vote_timed_out=false
|
|
3843
3885
|
while [ "$i" -le "$n" ]; do
|
|
3844
3886
|
local raw rc v raw1="" rc1=0
|
|
3845
|
-
raw="$(_eval_engine_run "$promptfile" "$timeout" 2>/dev/null)"; rc=$?
|
|
3887
|
+
raw="$(_eval_isolated_run _eval_engine_run "$promptfile" "$timeout" 2>/dev/null)"; rc=$?
|
|
3846
3888
|
raw1="$raw"; rc1="$rc"
|
|
3847
3889
|
if [ "$rc" -eq 124 ]; then
|
|
3848
3890
|
v=INVALID
|
|
3891
|
+
vote_timed_out=true
|
|
3849
3892
|
else
|
|
3850
3893
|
v="$(_eval_routing_parse_action "$raw" "$nonce")"
|
|
3851
3894
|
[ "$model" = "unknown" ] && model="$(_eval_parse_model "$raw")"
|
|
@@ -3857,9 +3900,9 @@ function cmd_eval_routing { # same arg shape as cmd_eval, called only when --sta
|
|
|
3857
3900
|
echo " $fid: retrying vote $i after ${backoff}s (parse-INVALID on attempt 1)"
|
|
3858
3901
|
sleep "$backoff" 2>/dev/null || true
|
|
3859
3902
|
fi
|
|
3860
|
-
raw="$(_eval_engine_run "$promptfile" "$timeout" 2>/dev/null)"; rc=$?
|
|
3903
|
+
raw="$(_eval_isolated_run _eval_engine_run "$promptfile" "$timeout" 2>/dev/null)"; rc=$?
|
|
3861
3904
|
retries_sum=$((retries_sum + 1))
|
|
3862
|
-
if [ "$rc" -eq 124 ]; then v=INVALID; else v="$(_eval_routing_parse_action "$raw" "$nonce")"; fi
|
|
3905
|
+
if [ "$rc" -eq 124 ]; then v=INVALID; vote_timed_out=true; else v="$(_eval_routing_parse_action "$raw" "$nonce")"; fi
|
|
3863
3906
|
[ "$model" = "unknown" ] && [ "$rc" -ne 124 ] && model="$(_eval_parse_model "$raw")"
|
|
3864
3907
|
local rl2; rl2="$(_eval_parse_rate_limited "$raw")"
|
|
3865
3908
|
[ "$rl2" = "true" ] && vote_rate_limited=true
|
|
@@ -3897,7 +3940,7 @@ function cmd_eval_routing { # same arg shape as cmd_eval, called only when --sta
|
|
|
3897
3940
|
echo " $fid: MISS - expected '$fexpected' (hit=$hit_count miss=$miss_count invalid=$invalid_count)"
|
|
3898
3941
|
fi
|
|
3899
3942
|
|
|
3900
|
-
_eval_routing_emit_result "$nonce" "$fid" "$fexpected" "$verdict" "$match" "$hit_count" "$miss_count" "$invalid_count" "$n" "$model" "$batch_cliv" "$batch_flowv" "$batch_rulessha" "$retries_sum" "$vote_rate_limited"
|
|
3943
|
+
_eval_routing_emit_result "$nonce" "$fid" "$fexpected" "$verdict" "$match" "$hit_count" "$miss_count" "$invalid_count" "$n" "$model" "$batch_cliv" "$batch_flowv" "$batch_rulessha" "$retries_sum" "$vote_rate_limited" "$psha" "$vote_timed_out"
|
|
3901
3944
|
n_written=$((n_written + 1))
|
|
3902
3945
|
|
|
3903
3946
|
if [ "$total_evaluated" -eq 1 ] && [ "$verdict" = "UNRELIABLE" ] && [ "$keep_going" -eq 0 ]; then
|
|
@@ -4005,15 +4048,16 @@ _eval_converge_parse_verdict() { # $1=raw $2=nonce -> GAP|CONVERGED|INVALID
|
|
|
4005
4048
|
|
|
4006
4049
|
_eval_converge_emit_result() { # same field shape as _eval_routing_emit_result
|
|
4007
4050
|
local run_id="$1" fid="$2" expected="$3" verdict="$4" match="$5" hit="$6" miss="$7" invalid="$8" n="$9"
|
|
4008
|
-
local model="${10}" cliv="${11}" flowv="${12}" rulessha="${13}" retries="${14:-0}" ratel="${15:-false}"
|
|
4051
|
+
local model="${10}" cliv="${11}" flowv="${12}" rulessha="${13}" retries="${14:-0}" ratel="${15:-false}" psha="${16:-}" timedout="${17:-false}"
|
|
4009
4052
|
case "$retries" in ''|*[!0-9]*) retries=0 ;; esac
|
|
4010
4053
|
case "$ratel" in true|false) : ;; *) ratel=false ;; esac
|
|
4054
|
+
case "$timedout" in true|false) : ;; *) timedout=false ;; esac
|
|
4011
4055
|
mkdir -p "$LOG_DIR" 2>/dev/null || true
|
|
4012
|
-
printf '{"ts":"%s","epoch_s":%s,"run_id":"%s","fixture":"%s","expected":"%s","verdict":"%s","match":"%s","votes":{"hit":%s,"miss":%s,"invalid":%s},"n":%s,"cli_version":"%s","model":"%s","flow_version":"%s","rules_sha":"%s","retries":%s,"rate_limited":%s}\n' \
|
|
4056
|
+
printf '{"ts":"%s","epoch_s":%s,"run_id":"%s","fixture":"%s","expected":"%s","verdict":"%s","match":"%s","votes":{"hit":%s,"miss":%s,"invalid":%s},"n":%s,"cli_version":"%s","model":"%s","flow_version":"%s","rules_sha":"%s","retries":%s,"rate_limited":%s,"prompt_sha":"%s","timed_out":%s}\n' \
|
|
4013
4057
|
"$(date -u +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || echo unknown)" "$(_now)" "$(_json_str "$run_id")" "$(_json_str "$fid")" \
|
|
4014
4058
|
"$(_json_str "$expected")" "$(_json_str "$verdict")" "$(_json_str "$match")" \
|
|
4015
4059
|
"$hit" "$miss" "$invalid" "$n" "$(_json_str "$cliv")" "$(_json_str "$model")" \
|
|
4016
|
-
"$(_json_str "$flowv")" "$(_json_str "$rulessha")" "$retries" "$ratel" >> "$EVAL_CONVERGE_RESULTS_FILE" 2>/dev/null || true
|
|
4060
|
+
"$(_json_str "$flowv")" "$(_json_str "$rulessha")" "$retries" "$ratel" "$(_json_str "$psha")" "$timedout" >> "$EVAL_CONVERGE_RESULTS_FILE" 2>/dev/null || true
|
|
4017
4061
|
}
|
|
4018
4062
|
|
|
4019
4063
|
_eval_converge_emit_batch_marker() { # $1=run_id $2=start|done $3=n
|
|
@@ -4075,7 +4119,9 @@ function cmd_eval_converge { # same arg shape as cmd_eval, called only when --st
|
|
|
4075
4119
|
return 1
|
|
4076
4120
|
fi
|
|
4077
4121
|
|
|
4078
|
-
|
|
4122
|
+
_eval_isolated_guard || return 1
|
|
4123
|
+
|
|
4124
|
+
local probe; probe="$(_eval_isolated_run _eval_probe)"
|
|
4079
4125
|
case "$probe" in
|
|
4080
4126
|
absent) echo "SKIP: 'claude' CLI not found on PATH - converge eval needs it to run the LLM judge. Zero calls made."; return 0 ;;
|
|
4081
4127
|
fail) echo "SKIP: 'claude' CLI present but the sentinel probe did not come back clean (one minimal probe call was made). Retry once 'claude -p' runs headless."; return 0 ;;
|
|
@@ -4123,17 +4169,20 @@ function cmd_eval_converge { # same arg shape as cmd_eval, called only when --st
|
|
|
4123
4169
|
echo " $fid: FAIL - could not build converge prompt (converge.md/repo-dir/challenge missing)"; total_miss=$((total_miss + 1)); rm -rf "$rundir" 2>/dev/null; continue
|
|
4124
4170
|
fi
|
|
4125
4171
|
|
|
4172
|
+
local psha
|
|
4173
|
+
psha="$(_eval_prompt_sha "$promptfile")"
|
|
4174
|
+
|
|
4126
4175
|
total_evaluated=$((total_evaluated + 1))
|
|
4127
|
-
local hit_count=0 miss_count=0 invalid_count=0 i=1 model="unknown" retries_sum=0 vote_rate_limited=false
|
|
4176
|
+
local hit_count=0 miss_count=0 invalid_count=0 i=1 model="unknown" retries_sum=0 vote_rate_limited=false vote_timed_out=false
|
|
4128
4177
|
while [ "$i" -le "$n" ]; do
|
|
4129
4178
|
local raw rc v raw1="" rc1=0
|
|
4130
|
-
raw="$(_eval_engine_run "$promptfile" "$timeout" 2>/dev/null)"; rc=$?; raw1="$raw"; rc1="$rc"
|
|
4131
|
-
if [ "$rc" -eq 124 ]; then v=INVALID; else v="$(_eval_converge_parse_verdict "$raw" "$nonce")"; [ "$model" = "unknown" ] && model="$(_eval_parse_model "$raw")"; fi
|
|
4179
|
+
raw="$(_eval_isolated_run _eval_engine_run "$promptfile" "$timeout" 2>/dev/null)"; rc=$?; raw1="$raw"; rc1="$rc"
|
|
4180
|
+
if [ "$rc" -eq 124 ]; then v=INVALID; vote_timed_out=true; else v="$(_eval_converge_parse_verdict "$raw" "$nonce")"; [ "$model" = "unknown" ] && model="$(_eval_parse_model "$raw")"; fi
|
|
4132
4181
|
local rl1; rl1="$(_eval_parse_rate_limited "$raw1")"; [ "$rl1" = "true" ] && vote_rate_limited=true
|
|
4133
4182
|
if [ "$v" = "INVALID" ] && [ "$rl1" != "true" ] && [ "$rc1" -ne 124 ]; then
|
|
4134
4183
|
if [ "$backoff" -gt 0 ]; then echo " $fid: retrying vote $i after ${backoff}s (parse-INVALID on attempt 1)"; sleep "$backoff" 2>/dev/null || true; fi
|
|
4135
|
-
raw="$(_eval_engine_run "$promptfile" "$timeout" 2>/dev/null)"; rc=$?; retries_sum=$((retries_sum + 1))
|
|
4136
|
-
if [ "$rc" -eq 124 ]; then v=INVALID; else v="$(_eval_converge_parse_verdict "$raw" "$nonce")"; fi
|
|
4184
|
+
raw="$(_eval_isolated_run _eval_engine_run "$promptfile" "$timeout" 2>/dev/null)"; rc=$?; retries_sum=$((retries_sum + 1))
|
|
4185
|
+
if [ "$rc" -eq 124 ]; then v=INVALID; vote_timed_out=true; else v="$(_eval_converge_parse_verdict "$raw" "$nonce")"; fi
|
|
4137
4186
|
[ "$model" = "unknown" ] && [ "$rc" -ne 124 ] && model="$(_eval_parse_model "$raw")"
|
|
4138
4187
|
local rl2; rl2="$(_eval_parse_rate_limited "$raw")"; [ "$rl2" = "true" ] && vote_rate_limited=true
|
|
4139
4188
|
fi
|
|
@@ -4158,7 +4207,7 @@ function cmd_eval_converge { # same arg shape as cmd_eval, called only when --st
|
|
|
4158
4207
|
total_miss=$((total_miss + 1)); echo " $fid: MISS - expected '$fexpected' (hit=$hit_count miss=$miss_count invalid=$invalid_count)"
|
|
4159
4208
|
fi
|
|
4160
4209
|
|
|
4161
|
-
_eval_converge_emit_result "$nonce" "$fid" "$fexpected" "$verdict" "$match" "$hit_count" "$miss_count" "$invalid_count" "$n" "$model" "$batch_cliv" "$batch_flowv" "$batch_rulessha" "$retries_sum" "$vote_rate_limited"
|
|
4210
|
+
_eval_converge_emit_result "$nonce" "$fid" "$fexpected" "$verdict" "$match" "$hit_count" "$miss_count" "$invalid_count" "$n" "$model" "$batch_cliv" "$batch_flowv" "$batch_rulessha" "$retries_sum" "$vote_rate_limited" "$psha" "$vote_timed_out"
|
|
4162
4211
|
n_written=$((n_written + 1))
|
|
4163
4212
|
|
|
4164
4213
|
if [ "$total_evaluated" -eq 1 ] && [ "$verdict" = "UNRELIABLE" ] && [ "$keep_going" -eq 0 ]; then
|
|
@@ -4297,7 +4346,7 @@ function cmd_eval {
|
|
|
4297
4346
|
|
|
4298
4347
|
local nonce="" total_mismatch=0 total_unreliable=0 total_evaluated=0 rundir="" n_written=0 aborted=0
|
|
4299
4348
|
# Computed ONCE per batch, not per fixture row - see _eval_emit_result's comment.
|
|
4300
|
-
local batch_cliv="" batch_flowv="" batch_grsha=""
|
|
4349
|
+
local batch_cliv="" batch_flowv="" batch_grsha="" prompt_sha_acc="" prompt_sha_enforced=0
|
|
4301
4350
|
if [ "$replay_mode" -eq 1 ]; then
|
|
4302
4351
|
# Replay wrap: skip probe SKIP-exit and _eval_cli_version only. Still compute
|
|
4303
4352
|
# batch_grsha (staleness hard-fail) and _flow_version. Nonce from fixture, never _eval_nonce.
|
|
@@ -4320,9 +4369,13 @@ function cmd_eval {
|
|
|
4320
4369
|
echo " recorded=$rec_sha current=$batch_grsha"
|
|
4321
4370
|
return 1
|
|
4322
4371
|
fi
|
|
4372
|
+
if grep -q '^prompt_sha\.' "$EVAL_REPLAY_DIR/meta" 2>/dev/null; then
|
|
4373
|
+
prompt_sha_enforced=1
|
|
4374
|
+
fi
|
|
4323
4375
|
echo "replay: keyless artifact batch (nonce from fixture, zero live calls)"
|
|
4324
4376
|
else
|
|
4325
|
-
|
|
4377
|
+
_eval_isolated_guard || return 1
|
|
4378
|
+
local probe; probe="$(_eval_isolated_run _eval_probe)"
|
|
4326
4379
|
case "$probe" in
|
|
4327
4380
|
absent)
|
|
4328
4381
|
echo "SKIP: 'claude' CLI not found on PATH - eval needs it to run the LLM judge. Zero calls made."
|
|
@@ -4425,13 +4478,36 @@ function cmd_eval {
|
|
|
4425
4478
|
continue
|
|
4426
4479
|
fi
|
|
4427
4480
|
|
|
4481
|
+
local psha
|
|
4482
|
+
psha="$(_eval_prompt_sha "$promptfile")"
|
|
4483
|
+
if [ "$replay_mode" -eq 1 ] && [ "$prompt_sha_enforced" -eq 1 ]; then
|
|
4484
|
+
local rec_psha
|
|
4485
|
+
rec_psha="$(_eval_replay_meta_get "prompt_sha.$fid")" || rec_psha=""
|
|
4486
|
+
if [ -z "$rec_psha" ] || [ "$rec_psha" != "$psha" ]; then
|
|
4487
|
+
echo "eval --replay: fixtures stale — prompt assembly changed; re-record live per ADR re-baseline rule"
|
|
4488
|
+
echo " recorded=$rec_psha current=$psha fixture=$fid"
|
|
4489
|
+
rm -rf "$rundir" 2>/dev/null
|
|
4490
|
+
trap - INT TERM
|
|
4491
|
+
return 1
|
|
4492
|
+
fi
|
|
4493
|
+
fi
|
|
4494
|
+
if [ "$record_mode" -eq 1 ]; then
|
|
4495
|
+
prompt_sha_acc="${prompt_sha_acc}prompt_sha.${fid}=${psha}"$'\n'
|
|
4496
|
+
fi
|
|
4497
|
+
|
|
4428
4498
|
total_evaluated=$((total_evaluated + 1))
|
|
4429
|
-
local flag_count=0 pass_count=0 invalid_count=0 i=1 model="unknown" retries_sum=0 vote_rate_limited=false
|
|
4499
|
+
local flag_count=0 pass_count=0 invalid_count=0 i=1 model="unknown" retries_sum=0 vote_rate_limited=false vote_timed_out=false
|
|
4430
4500
|
# Filename-safe fixture id for raw-capture paths - manifest fid is only tr -d '\r'-cleaned
|
|
4431
4501
|
# (the v1 trust-boundary is READ-side only, but we are now WRITING keyed by it), so a
|
|
4432
4502
|
# hand-edited or FLOW_EVAL_MANIFEST-overridden manifest cannot traverse out of eval-raw/.
|
|
4433
4503
|
local fid_safe; fid_safe="$(printf '%s' "$fid" | tr -c 'A-Za-z0-9' '-' | sed -E 's/-+/-/g; s/^-//; s/-$//')"
|
|
4434
4504
|
[ -z "$fid_safe" ] && fid_safe="_"
|
|
4505
|
+
if [ "$replay_mode" -eq 0 ]; then
|
|
4506
|
+
local rawdir_prompt="$LOG_DIR/eval-raw/$nonce"
|
|
4507
|
+
if mkdir -p "$rawdir_prompt" 2>/dev/null; then
|
|
4508
|
+
cp "$promptfile" "$rawdir_prompt/${fid_safe}.prompt" 2>/dev/null || true
|
|
4509
|
+
fi
|
|
4510
|
+
fi
|
|
4435
4511
|
while [ "$i" -le "$n" ]; do
|
|
4436
4512
|
local raw rc v raw1="" rc1=0 err1="" err2=""
|
|
4437
4513
|
# Attempt 1 - stderr captured at the OUTER redirection so it never enters the sh -c
|
|
@@ -4442,11 +4518,12 @@ function cmd_eval {
|
|
|
4442
4518
|
raw="$(_eval_replay_vote "$fid" "$i")" || { rm -rf "$rundir" 2>/dev/null; trap - INT TERM; return 1; }
|
|
4443
4519
|
rc=0
|
|
4444
4520
|
else
|
|
4445
|
-
raw="$(_eval_engine_run "$promptfile" "$timeout" 2>"$err1")"; rc=$?
|
|
4521
|
+
raw="$(_eval_isolated_run _eval_engine_run "$promptfile" "$timeout" 2>"$err1")"; rc=$?
|
|
4446
4522
|
fi
|
|
4447
4523
|
raw1="$raw"; rc1="$rc"
|
|
4448
4524
|
if [ "$rc" -eq 124 ]; then
|
|
4449
4525
|
v=INVALID
|
|
4526
|
+
vote_timed_out=true
|
|
4450
4527
|
else
|
|
4451
4528
|
v="$(_eval_parse_verdict "$raw" "$nonce")"
|
|
4452
4529
|
[ "$model" = "unknown" ] && model="$(_eval_parse_model "$raw")"
|
|
@@ -4477,10 +4554,10 @@ function cmd_eval {
|
|
|
4477
4554
|
raw="$(_eval_replay_vote "$fid" "$i")" || { rm -rf "$rundir" 2>/dev/null; trap - INT TERM; return 1; }
|
|
4478
4555
|
rc=0
|
|
4479
4556
|
else
|
|
4480
|
-
raw="$(_eval_engine_run "$promptfile" "$timeout" 2>"$err2")"; rc=$?
|
|
4557
|
+
raw="$(_eval_isolated_run _eval_engine_run "$promptfile" "$timeout" 2>"$err2")"; rc=$?
|
|
4481
4558
|
fi
|
|
4482
4559
|
retries_sum=$((retries_sum + 1))
|
|
4483
|
-
if [ "$rc" -eq 124 ]; then v=INVALID; else v="$(_eval_parse_verdict "$raw" "$nonce")"; fi
|
|
4560
|
+
if [ "$rc" -eq 124 ]; then v=INVALID; vote_timed_out=true; else v="$(_eval_parse_verdict "$raw" "$nonce")"; fi
|
|
4484
4561
|
[ "$model" = "unknown" ] && [ "$rc" -ne 124 ] && model="$(_eval_parse_model "$raw")"
|
|
4485
4562
|
local rl2; rl2="$(_eval_parse_rate_limited "$raw")"
|
|
4486
4563
|
[ "$rl2" = "true" ] && vote_rate_limited=true
|
|
@@ -4555,7 +4632,7 @@ function cmd_eval {
|
|
|
4555
4632
|
fi
|
|
4556
4633
|
|
|
4557
4634
|
if [ "$replay_mode" -eq 0 ]; then
|
|
4558
|
-
_eval_emit_result "$nonce" "$fid" "$fstage" "$fexpected" "$verdict" "$match" "$flag_count" "$pass_count" "$invalid_count" "$n" "$model" "$batch_cliv" "$batch_flowv" "$batch_grsha" "$retries_sum" "$vote_rate_limited"
|
|
4635
|
+
_eval_emit_result "$nonce" "$fid" "$fstage" "$fexpected" "$verdict" "$match" "$flag_count" "$pass_count" "$invalid_count" "$n" "$model" "$batch_cliv" "$batch_flowv" "$batch_grsha" "$retries_sum" "$vote_rate_limited" "$psha" "$vote_timed_out"
|
|
4559
4636
|
fi
|
|
4560
4637
|
n_written=$((n_written + 1))
|
|
4561
4638
|
|
|
@@ -4587,7 +4664,7 @@ function cmd_eval {
|
|
|
4587
4664
|
_eval_emit_batch_marker "$nonce" done "$n_written"
|
|
4588
4665
|
fi
|
|
4589
4666
|
if [ "$record_mode" -eq 1 ] && [ "$aborted" -eq 0 ]; then
|
|
4590
|
-
_eval_replay_write_meta "$nonce" "$batch_grsha" "${model:-unknown}" "$n"
|
|
4667
|
+
_eval_replay_write_meta "$nonce" "$batch_grsha" "${model:-unknown}" "$n" "$prompt_sha_acc"
|
|
4591
4668
|
fi
|
|
4592
4669
|
|
|
4593
4670
|
echo
|
package/skills-manifest.json
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
{
|
|
2
2
|
"source": "../skills/flow",
|
|
3
3
|
"destRelative": "skills/flow",
|
|
4
|
-
"fileCount":
|
|
5
|
-
"syncedAt": "2026-08-
|
|
4
|
+
"fileCount": 103,
|
|
5
|
+
"syncedAt": "2026-08-20T13:35:30.768Z",
|
|
6
6
|
"fileList": [
|
|
7
7
|
"SKILL.md",
|
|
8
8
|
"_templates/00-idea.md",
|
|
@@ -97,6 +97,7 @@
|
|
|
97
97
|
"references/gate-eval.md",
|
|
98
98
|
"references/gate-rules.md",
|
|
99
99
|
"references/ground-truth-gates.md",
|
|
100
|
+
"references/host-agnostic-parallel.md",
|
|
100
101
|
"references/loop-harness-2026-principles.md",
|
|
101
102
|
"references/mode-work.md",
|
|
102
103
|
"references/native-rituals.md",
|