flow-workflows 0.65.0 → 0.67.0

package/plugins/flow/CHANGELOG.md

@@ -5,6 +5,233 @@ plugin and is what `/flow:news` reads to show you what changed since your previo
 
 The canonical, richest notes live in the [GitHub Releases](https://github.com/mashware/flow-workflows/releases).
 
+## v0.67.0 — Every layer of the review was argued from judgement, and none was ever checked  ·  2026-09-21
+
+**In short**
+- **The review now has a bench.** `plugins/flow/evals/` runs `/flow:feat:review` over fixtures whose outcome is known before the review starts, and scores what came back: eight cases, five with a defect seeded at a known `file:line` the review has to raise, three with a clean diff where any blocker it reports is a false positive.
+- **The seeded defects come from this repository's own history**, not from imagination — a value recorded after the mutation instead of before, a failing call reported as a clean result, a filter that drops the item it exists to carry, several writers on one shared key, a threshold that is code in one place and a literal in another. A defect we invent is a defect shaped like something we were already looking for.
+- **Deterministic graders carry the score.** No judge model decides anything: a judge introduces the variance the bench exists to measure, and it changes the measuring stick exactly when the model changes. The `llm` grader is not used, and the two artifact checks that only the plugin can pass are marked as indicators outside the score, so the no-plugin arm stays comparable.
+- **A difference inside the base's own range is not a difference.** `script/bench-compare.py` reads two runs, reports the **median** of each case rather than the mean the summary prints, and prints `—` for anything that lands inside the range the base's own runs spanned. A bench that reports every wobble as a regression is one everybody learns to ignore.
+- **zcode is a harness the plugin knows.** It reads this same plugin package, so the commands were already there — but under names that collide with its built-ins, and resolving Claude Code's overlay. It now has its own `/flow:`-prefixed command set generated into the package, and `FLOW.zcode.md` is a recognised overlay everywhere, the CLI included.
+- **Baselines are versioned and committed**, one per plugin version and model, under `evals/baselines/`. When a new model ships, the suite runs with `--model` and everything else frozen — which is the only way to find out that a layer which used to help has started costing turns.
+
+Nothing in this repository could answer whether a change to `review` found more defects, fewer, or
+the same ones more expensively. `script/check.py` proves the adapters match their sources byte for
+byte and that a cited path exists; the hook tests pin behaviour that is deterministic by
+construction. All of it checks **shape**, and the README already conceded half the gap — *"Cost
+figures in these docs are estimates, not measurements"* — while saying nothing at all about the
+other half.
+
+That gap was not abstract. The completeness sweep, the data-access duel, the idiom audit, the
+150/600/1500 thresholds, the sensitive-surface bump: every one of them was argued from judgement,
+and every one is now remembered as conviction. v0.66.0 wrote down the rule for retiring a pass —
+no exclusive surviving finding across twenty reviews and it comes out of the command — and there
+was no way to run twenty comparable reviews to trigger it. The same release made the tier
+computable and nothing could say whether the tier it computed was the right one. Both shipped
+instrumentation with no bench to read it on.
+
+**A case is a repository, not a prompt.** Each one scaffolds a small tree, a flow work parked at
+`build`, and a branch whose diff has a known outcome; the flow CLI is installed into the fixture
+from this checkout, so the round takes the CLI path and not its documented fallback. The prompt
+opens with the literal `/flow:feat:review` and then states the task in plain words — both halves
+are load-bearing. Inside an eval run the agent cannot invoke a plugin command on its own: asked in
+prose to use the review command it reads files instead and never reaches it, so only a prompt that
+*is* the command gets there. The plain-words half is what the no-plugin arm answers, which is what
+makes `Δ` mean anything.
+
+**Both kinds of case are mandatory**, and the second kind is the one that is easy to skip. Without
+a clean diff in the suite the score rewards the noisiest review there is, and noise is what makes
+people stop reading reviews at all. One of the three sits on a path `quality.sensitive_paths` names
+and moves a log level and nothing else — which is the judgement §2.0 says a CLI cannot make, checked
+for the first time.
+
+The release preflight gains the checks that keep the bench runnable — every case has its scaffold,
+its prompt and its graders, the prompt still opens with the command under test, and the suite still
+holds both kinds. Running it is deliberately **not** wired into `publish.yml`: that workflow puts
+the package on npm in minutes, and ten cases at three runs across two arms is sixty full agent
+sessions. `.github/workflows/bench.yml` runs at the release and on demand, and says so plainly when
+the repository has no credential for it rather than standing red.
+
+**What the first run said**, recorded here because a bench whose first result is left out of the
+notes is a bench nobody will believe later. 8 cases × 3 runs × 2 arms, 48 sessions, 52 minutes,
+$47.27, threshold 0.5, the CLI's default model: every case at or above the threshold, suite score
+0.96, and **mean Δ against the no-plugin arm of −0.04**. Seven of the eight cases scored 1.00 in
+both arms; the eighth (`seeded-divergent-threshold`) scored 0.67 with the plugin and 1.00 without,
+one run in three having missed it. The with-arm cost $1.57 and 4.5 minutes per run against $0.40
+and 1.9 minutes — **3.9× for the same findings**.
+
+The honest reading is about the fixtures before it is about the command, and the reason is
+mechanical: every case is tagged `xs`, and **at XS the command runs the built-in `code-review`
+alone** — no panel, and §3.5's sweep is M/L only. The layers this bench exists to judge did not run
+once in those 48 sessions, so the two arms were doing nearly the same work. Eight single-file diffs
+with a docstring stating the contract the code breaks is also a defect an unguided review finds,
+which puts both arms on the ceiling, where a delta can only fall.
+
+The second tier is written down rather than written: `deep` cases, M-sized, with a populated
+reviewer roster, a defect split across files and its contract in `03-design.md`. They cost roughly
+what they exercise, so the suite splits by tag and each tier keeps its own baseline.
+
+Nothing is retired and no threshold moves here. The first run establishes the base; it does not
+judge anything.
+
+**zcode**, separately. It reads Claude-format plugin packages — `.zcode-plugin/plugin.json` first,
+then `.claude-plugin/plugin.json` — and exposes the same primitives under the same names, so
+installing flow from the marketplace worked on the first try and hid two things. A command is named
+after its path with no plugin prefix, so the set arrived as `/feat:start`, `/next`, `/init` and
+`/doctor` — the last two losing to zcode's own built-ins. And every `flow-workflows` call in the
+pages carries `--harness claude`, which on zcode reads another product's overlay and reports, in the
+pack's own header, that an overlay was read.
+
+Both are fixed by generation, not by prose: `plugins/flow/zcode-commands/flow/**` is the same
+generator's fifth target, the `flow/` folder is what makes zcode name them `/flow:feat:start`, and
+the flag is substituted like it is for every other mirror. `FLOW.zcode.md` joins the overlay names
+in flow-core, `init`, the template and the CLI — where an unknown `--harness` has always stopped
+the run rather than silently reading the base. The plugin's own Claude pages still show up
+unprefixed in zcode, so both the mirror's legend and the docs say to type the `/flow:` names.
+
+## v0.66.0 — Every phase read a tree, and none of them could say which one  ·  2026-09-21
+
+**In short**
+- **A round that contains a writer is not one round.** An agent whose deliverable is files on disk now holds the working tree alone: readers still fan out freely, writers go one at a time, and the parent's own edits wait until the round is consolidated. `validate` runs its testing agent before the suite and the performance pass rather than beside them; a review round reads a tree that has stopped moving.
+- **`ship` checks the checkout it did not push.** With `meta.json.worktree` set, the Close reads the main checkout's `git status` and names, in one line, anything sitting there — then offers once to carry it onto the branch. A configured tool that wrote to the wrong tree stops being invisible.
+- **A review freezes what it is about to read.** `review` resolves one revision before it launches anything — `HEAD` over a clean tree, `git stash create` over a dirty one — and `flow bundle --rev` makes the pack that revision for every reader. The sha recorded at the end is the tree that was **read**, with the round's own fixes recorded separately as `fixed_sha`, so `ship` asks about them instead of reporting a match.
+- **A re-review reads the delta, and is sized by it.** The previous `reviewed_sha` becomes the review's own base when it is an ancestor of the candidate. A 30-line delta on a 900-line branch stops being reviewed as the 900, and the two commands that already promised a scoped re-review now get one.
+- **The implementation log names the commit of every step.** `05-implementation.md` gains a `## Steps` section — one line per step with its short sha, `—` when the step was not committed. The hot cut reads its cut point there and proposes a sha instead of a description.
+- **The review tier is computed, not narrated.** `flow tier` resolves the depth from the config the round already reads — changed lines, effective size, the effort with and without the sensitive-surface bump, the ceilings, the within-10% flag, the `sensitive_paths` that matched — and §2.0 shrinks to the judgement a command cannot make.
+- **A finding says which pass found it.** Dedup keeps every source rather than the first, `meta.json.review_findings[]` records them, and `flow cost` reports per origin how many findings each pass raised **alone** and how many of those survived verification. The rule for retiring a pass is written down before the data arrives.
+
+Four failures with one shape: a phase read a tree, and nothing in the flow could say which tree
+that was. Two of them came out of the same real run and could not be seen by reading the plugin;
+the other two were visible on the page all along, in a sha written at the wrong moment.
+
+**The tree had no owner.** `validate §2` launched its three pieces in parallel, and one of them
+writes: the testing agent's deliverable is test files, saved as each is finished. The other two
+read — the full suite, and the performance pass — and the main thread was editing alongside all of
+them. One suite run came back red for a mutation belonging to another agent. A reviewer found a
+source file changed without it being its doing, and had to run `git status` around each of its own
+findings to know which were real. The commit taken at the end was clean by luck.
+
+The fix is a rule in `flow-core §6` rather than a fix in `validate`, because the shape repeats
+wherever a phase fans out: a writing agent is break, run, restore, and it composes with nothing
+else looking at the same tree. Readers are unaffected and stay as parallel as they ever were —
+which is most rounds. Writers are serialised or given a worktree of their own, and the readers sit
+entirely before them or entirely after. `validate` now runs the testing agent first and alone, and
+that ordering pays for itself twice: the suite that follows is the first thing that has ever
+executed the tests the phase just added. `review` states the other half of the same rule — this
+phase's own fixes land after the round is consolidated, never underneath reviewers still reading.
+Ordering costs wall clock, the phase says so, and a reading nobody can trust costs the phase twice.
+
+**The second tree was the one nobody looked at.** A work living in a worktree ran `ship §5` and
+consolidated five knowledge cards through the configured `knowledge.save` tool. They were written
+into the *main* checkout, which was standing on the base branch — not into the branch about to be
+pushed. The MR went out without them, and the session noticed on its own and carried the files
+across by hand.
+
+The tool did nothing wrong: it is started once per session, resolves its store from the directory
+it was started in, and its call takes no root, so nothing in the invocation could have corrected
+it. flow is the only party that knows `meta.json.worktree` is set. So `ship` now has a §5.5 that
+reads the main checkout's `git status --porcelain` before the Close, names any paths it finds as
+what they are — in the main checkout, not in the branch that was pushed, and not necessarily put
+there by this run — and offers once, in every autonomy mode, to copy them over, commit and push,
+which updates the MR/PR that already exists. Leaving them is a legitimate answer and the Close
+reports either way: a `ship` that silently repaired two checkouts is worse than one that named the
+problem. `bug:ship` runs the same check, where it catches what `postmortem` consolidated one phase
+earlier.
+
+`knowledge.save` was merely the first tool to fall in; no key is special, and the check names none.
+The one write flow does control was corrected at the source: the `KNOWLEDGE.md` fallback said "the
+repo root", which `flow-core §0` defines as the main checkout's — correct for reading a git-ignored
+config and exactly backwards for a tracked file, which belongs to the branch. It is written to the
+checkout the phase is running in.
+
+**The review recorded the tree it ended on, not the one it read.** `review`'s scope was the working
+tree, and the only sha it wrote was `git rev-parse HEAD` at its Close — taken after the round had
+applied its own fixes, which the README's own first-minute example advertises. So the tree the gate
+remembers contains, by construction, code no reviewer read; `ship` compares that sha against `HEAD`,
+finds the equality it was built to detect a delta in, and continues without a line. A round that
+fixed findings without committing was worse still: two identical shas over two different trees.
+
+`review §1.5` resolves the candidate before §2 launches anything — `git rev-parse HEAD` when the
+tree is clean, `git stash create` when it is dirty (a real commit object holding the index and the
+tracked tree, the checkout untouched, nothing to pop) — and anchors it at
+`refs/flow/candidate/<TICKET>`, because an unreferenced object the `ship` gate can no longer resolve
+is a gate that reports nothing. `flow bundle --rev` reads that revision for the diff *and* for the
+file contents; getting only one of the two from the revision would hand a reviewer a diff of one
+tree and the files of another, which looks right in both halves. Untracked files are in no revision
+at all, so they are named rather than quietly missed, and `quality.static_analysis` still runs
+against the checkout and says so. The Close then writes two fields, because they are two facts:
+`reviewed_sha` is what was read, `fixed_sha` is `HEAD` after the round's own fixes or empty when it
+made none. `ship` gained three cases it could not express before — `fixed_sha` set alongside an
+equal sha is a mismatch, a dirty tree at either end is named instead of passed over, and a
+`reviewed_sha` that no longer resolves is reported rather than read as "nothing wrong here".
+
+**And the same sha, once it was trustworthy, had a second job.** `ship` offered *"re-review the
+delta"* and `respond` promised a review *"scoped to this round's diff"* — and the command they both
+call declared the opposite, reading the full branch and resolving its tier against the accumulated
+total. That is the wasted-review failure §2.0 already names, arriving on the time axis instead of
+the train axis: thirty new lines reviewed with the depth nine hundred lines earn. `review §1.5` now
+takes the previous `reviewed_sha` as its base when it is non-empty and an ancestor of the candidate,
+and `git.default_base` otherwise, saying which in one line. A first review has no `reviewed_sha` and
+reads the branch exactly as before; `--full` forces that on demand. Two cases fall back and say why,
+because a delta cannot see how it interacts with what was cleared before: it touches a file a
+previous finding sat on, or it changes an external contract. The sensitive-surface bump never scales
+away — twelve lines of an authorization check still get the panel.
+
+**And the log of what was built named not one commit that built it.** `build` commits one WIP per
+step in `guided`/`auto`, and nothing wrote down which commit belonged to which step:
+`05-implementation.md` is the running log of the build and held no sha at all. So the hot cut asked
+to *"identify a cut point: the last WIP commit where the piece is coherent"* — read off
+`git log --oneline` and matched by the prose in `WIP <TICKET>: <step>` — the transfer that follows
+it picked its commits the same way, and a delta-scoped review could say which lines were new but
+not which step they came from. The step summary was already being written for every step; the sha
+is the one identifier that makes the log line and the git history the same record. It is now a
+`## Steps` section, one line each, written in every autonomy mode with no question, and a step that
+was **not** committed gets its line too with `—` where the sha would be — the absence of a commit
+is exactly what a reader of that log needs to see. The cut proposal names the sha it is cutting at.
+
+**The budget was code and the tier was still prose.** `v0.65.0` moved most of a round into the CLI
+— `bundle` assembles the material, `review` runs the panel, `cost` reads back what it spent, the
+`budget_max`/`fanout_max` cap is applied in code. §2.0 did not move: 822 words, the longest section
+of a 5 683-word file, instructing a model to measure `git diff --shortstat`, read a four-row table,
+take the lower of that and `meta.json.size`, apply the bump, and flag a diff landing within 10%
+under a threshold. Every step of it arithmetic, none of it judgement — and the real cost was not
+the tokens. `06-review.md` recorded `Review tier: … per §2.0`, which is the tier the agent *said*
+it derived: a round that eyeballed the diff and a round that measured it write a byte-identical
+artifact, on the one number in the command that the section itself says carries an incentive.
+
+`flow tier` prints it — changed lines over the reviewed range, diff size, `meta.json.size`,
+effective size, the built-in effort with and without the bump, whether the panel runs, the resolved
+ceilings, the within-10% flag, and which `quality.sensitive_paths` globs the diff matched with the
+path that matched each. Copied verbatim into §8, the way `bundle --checks` copies static analysis
+out. What it deliberately does **not** decide is what a command cannot see: whether a generic
+category applies (auth, secrets, payments, personal data, a public contract, a schema change), and
+whether a matched path is a control-flow change or observability only — a log level still gets the
+review its size earned. The sensitive list stays the repo's; the CLI reads the key and carries no
+idea of its own. No CLI or a non-zero exit falls back to the prose, kept in abbreviated form, with
+`Review path` naming the fallback. Twenty-five cases in `script/tests/tier.sh` pin the table at its
+boundaries — 150/151, 600/601, 1500/1501 — plus the lower-of-two rule, the misclassification note,
+the two `review_depth` short circuits, and that `--rev` makes the tier belong to the same revision
+the reviewers read.
+
+**And a finding did not say which pass found it, so no pass could ever be retired.** Beyond the
+built-in and the panel, a round runs four specialised passes — the completeness sweep, the
+data-access duel, the double-blind contract verification, the idiom audit — around 1 800 words of
+the command plus the agents each one launches. Every one was written after a real defect got
+through. Not one could show it is still earning its cost, because a finding that reached
+`## Blockers` had lost where it came from: one the built-in already had and one only the idiom
+audit saw were written identically, and after dedup a finding three sources raised looked exactly
+like a finding one source raised. The only available argument was an opinion, and opinions about
+this converge on whoever spoke last.
+
+Findings now carry an origin from §2.1 through to the artifact, **dedup keeps every source rather
+than the first**, and `meta.json.review_findings[]` records `{round, origin, severity, file,
+discarded_by_skeptic}` — machine-readable, no finding text, which stays in `06-review.md`.
+`flow cost` reads them back per origin: how many findings each pass contributed, how many **only**
+it raised, and how many of those survived §6. That last column is the one that decides anything.
+The retirement rule is written down before the data arrives so it cannot be argued backwards later
+— a pass with no exclusive surviving finding across twenty reviews comes out of the command. No
+pass is removed or weakened here; this only makes their contribution visible.
+
 ## v0.65.0 — Every phase rediscovered the same context, and nobody knew what it cost  ·  2026-09-21
 
 **In short**

package/plugins/flow/examples/FLOW.template.md

@@ -5,8 +5,8 @@ file in their step 0. Delete what does not apply; **empty or absent = auto-disco
 default behavior** (each command states what it does when a value is missing).
 
 Use this shape for the shared base `FLOW.md` and for an optional sparse harness overlay:
-`FLOW.claude.md`, `FLOW.codex.md`, `FLOW.opencode.md`, `FLOW.gemini.md`, or `FLOW.hermes.md`. The
-active overlay is
+`FLOW.claude.md`, `FLOW.codex.md`, `FLOW.opencode.md`, `FLOW.gemini.md`, `FLOW.hermes.md`, or
+`FLOW.zcode.md`. The active overlay is
 read after the base. A key present in it replaces the base value; a present-but-empty key masks the
 base and selects the normal empty fallback; an absent key inherits the base. Lists replace whole
 lists. `conventions` is additive: overlay lines follow the base lines and both apply. Existing repos