@surlymochan/evalid 0.0.0-stage → 0.4.1

package/CHANGELOG.md

@@ -0,0 +1,94 @@
+# Changelog
+
+All notable changes to this project are documented here. Detailed notes live in
+[RELEASE_NOTES.md](RELEASE_NOTES.md).
+
+## [Unreleased]
+
+## [0.4.1] - 2026-11-06
+
+### Fixed
+- **TDD gate counts skipped/todo tests again on Node 24**: the gate parsed skip/todo counts only from the TAP reporter's `# skipped / # todo` summary lines, but Node 24's built-in runner defaults to the spec reporter when piped, whose summary lines carry a `ℹ` marker — an all-skipped/todo run exited 0, parsed as zero skips, and was waved through as green. The parser now reads the runner's summary block in both reporter formats (anchored to the summary lines, gated on the full node signature), and fails closed: a node summary whose gate counts cannot be fully parsed classifies as `counts_unknown` and is rejected by proof validation (`tests_counts_unknown`) instead of reading as zero skipped.
+
+### Added
+- **Phase cost evidence on every run (issue #5 P1-1)**: the per-phase timing/token report (`src/core/phase-cost.js`, landed in 0.3.0) now also rides on the `workflow_failed` execution-log event — a run that stops mid-way still reports where its time and tokens went — and is persisted as a standalone artifact at `.as-xflow/phase-cost.json` on both completion and failure, so the breakdown is readable without mining the ndjson execution log. Acceptance rerun on the SWE-bench cells stays pending the benchmark environment; the accounting structure (phase sum vs wall clock within 5%) is covered by `test/phase-cost.test.js`.
+
+## [0.4.0] - 2026-11-04
+
+### Changed
+- **Overhead cut 1 (v0.4 F2, issue #5 P1-2/P1-4)**: the lite (yolo) track seals the TDD red proof immediately after `plan` — the issue/branch ceremony no longer sits between planning and red-proof, and the planning pass now authors the failing test, so plan and red share one agent pass. `xflow workflow run --title "..."` also derives the change id from the title when the branch cannot provide one (previously a hard stop on `main` that pushed agents into manual atom-by-atom driving). TDD snapshots are scoped: inside a git work tree I6a seals only the files that differ from HEAD plus the files the test command names, and I6b materializes the rest from the recorded HEAD at replay — on the SWE-bench sympy tree the sealed manifest drops from 1,543 files / 164 KB to the changed files only (2 files / 197 B for a one-file change), and the snapshot copy from 26.9 MB to ~51 KB. Proofs sealed before this change keep verifying (the scope joins the binding only when present).
+
+### Added
+- **CR gate `J6.cr.review` wired into the verify stage (v0.4 F1)**: closes the v0.3.0 known gap where J6 was registered and unit-tested but no workflow phase invoked it. The yolo `verify-consistency` phase and the corps `qa` phase now run J6 on every change — after the spec-delta review, before the stage flips to verify — and declare its verdict record `specs/changes/<change_id>/cr/review-0.json` as a required artifact. A failing verdict (spec misalignment, changed code with no test follow, net assertion decrease, blown loop budget) stops the workflow at verify instead of waving the change through to archive. Orchestration evidence: `test/cr-review-wiring.test.js` drives the real yolo verify phase through `WorkflowExecutor` against a git fixture — an aligned, tested change passes with a verdict record, and an untested change is stopped at the phase.
+- **SWE-bench predictions emitted directly by `run.py` (v0.4 F3)**: the bench harness no longer relies on a one-off converter outside the repo. A stdlib-only `bench/swebench/harness/predictions.py` builds records with exactly `instance_id`, `model_name_or_path`, `model_patch`, re-applies the P0-2 evidence filter (`specs/`, `.xflow/`, `.as-xflow/`) on both paths, and upserts per-arm files so resumable reruns never duplicate lines; `run.py --export-predictions` rebuilds the same files from saved patches with no model calls. Regression: all four pilot arms rebuilt through the new path are byte-identical to the hand-converted pilot files (`cmp` = 0 across 24 instances), and the largest openflow-arm patch stays at 9,650 bytes, well under the 200 KB line.
+- **`xflow init` writes the project `.gitignore` (v0.4 F4)**: init now creates or merges `.gitignore` entries for evalid local state (`.as-xflow/`, `specs/changes/`), appending only the entries that are missing and never modifying existing lines; `.xflow/` stays tracked on purpose (`GOAL.md` is a reviewable project document). Re-running `init --force` tops up projects initialized before v0.4. Closes the v0.3.0 known gap.
+
+## [0.3.1] - 2026-11-04
+
+### Changed
+- **Renamed: OpenFlow → evalid.** The repository is now `surlymochan/evalid` (old links redirect) and the npm package is now `evalid`. Same code and behaviour as 0.3.0; the rename matches the repositioned first screen — a runtime verification layer for coding agents ("Claimed ≠ verified"). The `xflow` CLI name is unchanged in this release.
+
+## [0.3.0] - 2026-11-04
+
+### Added
+- **SPEC clarification gate `J4d.spec_clarify.gate`** (#11): blocks ambiguous specs before the loop starts. Flags MUST clauses with no quantified anchor, missing WHEN/THEN scenarios, vague wording (EN/CN tables) and TBD/placeholder text; every finding carries an answerable clarification question. Available as `xflow spec clarify` and wired into the yolo/corps proposal-consistency stage.
+- **Code review gate `J6.cr.review`** (#10): deterministic review checks — spec alignment (zero alignment fails), test-follow (untested changes fail unless `cr_allow_untested` is declared in status), net assertion decrease fails, test files may not be edited to force green, and a loop budget (default 10 files / 400 lines; warn over budget, fail at 2x).
+- **Test gate v2 checks in `I6c`** (#12): rejects expected values sourced from the implementation itself, dead-callback assertions, and self-comparison tautologies. Adds per-phase cost accounting (`src/core/phase-cost.js`): wall-clock and token spend per phase, flagged when it diverges from total wall-clock by more than 5%.
+
+### Fixed
+- **Benchmark evidence no longer leaks into solution patches** (#8): `run.py` now excludes `specs/` when generating `solution.diff`, with a second filter in the harness (regrade strips it from old patches too). A real openflow-arm patch shrank from 17.0 MB to 24 KB — 99.9% of it was evidence files, not code.
+- **Per-run uv cache** (#8): `UV_CACHE_DIR` is set per run instead of sharing one cache across benchmark arms.
+- **CI runner pinned to `ubuntu-24.04`** (#9): Playwright 1.59.1 does not support chromium on ubuntu 26.04, which silently killed the verify job.
+
+### Docs
+- Added `CONTRIBUTING.md` and issue/PR templates (#8, #11, #10). Workflow manual atom count updated 83 → 85.
+
+### Known gaps
+- `J6.cr.review` is registered and tested but not yet wired into the yolo/corps verify stage (planned for v0.4).
+- `xflow init` does not yet add OpenFlow artefacts to the user project's `.gitignore` (planned for v0.4).
+
+## [0.2.2] - 2026-10-18
+
+### Docs
+- Added the SWE-bench Verified results to the README and `bench/swebench/`: 6 hard sympy, pytest and pylint instances on DeepSeek-V4.1-Flash (`deepseek-flash`), n=1 per cell.
+- Corrected the first published results after a pytest temp-directory race in grading: `pytest-5787` and `pytest-6197` became resolved for all arms; `pytest-10356` is unresolved by every arm. See the [regrade notes](bench/swebench/README.md#regrade-2026-10-18).
+- Linked the [optimization backlog issue #5](https://github.com/surlymochan/openflow/issues/5).
+
+No CLI or gate changes in this release.
+
+## [0.2.1] - 2026-10-17
+
+### Changed
+- TDD proofs are bound to a tree snapshot. `I6b` recomputes the binding and re-runs the recorded command. Green also checks that snapshotted test files still match disk.
+- Red and green must run the same test, or green must be a superset. Skip, todo, and focused tests do not count. Import-only or empty failures do not count as red. Type-only assertions fail when they are the only assertions.
+- `J4b` fails when the same subject has contradictory `MUST`/`SHALL` verb/value clauses.
+- `self.assert*`, `pytest.raises`, and bare `assert` count as assertions.
+- `tdd_mode` `refactor` and `docs` (status.json or `--tdd-mode`) are the exemption path for an honest refactor or a docs-only diff. Misuse fails.
+
+## [0.2.0] - 2026-10-17 (relaunch)
+
+### Added
+- Policy overlay execution mode: `xflow policy validate`, plus policy, notify, storage, plugin and
+  experience-digest atoms.
+- `xflow intake recommend` for adaptive yolo/corps track selection.
+- `xflow profile show|set` for execution profiles.
+- `xflow compare super-assistant`.
+- `docs/assets/demo.gif`, recorded from a real CLI run, and a README rewrite.
+- `CHANGELOG.md`.
+
+### Changed
+- The npm package is now named `@surlymochan/openflow`, because the unscoped `openflow` name is taken. The CLI is still `xflow`.
+- `package.json` adds repository/homepage/bugs/keywords/author, `engines.node >= 22.5` and public `publishConfig`.
+- The `xflow serve` fallback message no longer links to an unpublished repository.
+
+### Fixed
+- `agent_invoke` no longer crashes with an uncaught `EPIPE` when an agent CLI exits before reading its stdin prompt. This fixes an intermittent `heavy-atoms` test failure.
+
+## [0.1.0] - 2026-05-25
+
+### Added
+- First public extraction:
+  - YAML workflows (`yolo`, `corps`) validated against the atom registry.
+  - Red/green TDD proof gates and a test-quality review.
+  - Handoff and AHA state, archive ordering, and the `xflow` CLI.
+- Adapters: `xflow adapter import-file` and `xflow adapter github-issue`.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 evalid contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,316 @@
-# Temporary Holding Version
+# evalid
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+[![CI](https://github.com/surlymochan/evalid/actions/workflows/ci.yml/badge.svg)](https://github.com/surlymochan/evalid/actions/workflows/ci.yml)
+[![npm](https://img.shields.io/npm/v/@surlymochan/evalid?label=npm)](https://www.npmjs.com/package/@surlymochan/evalid)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+![node >= 22.5](https://img.shields.io/badge/node-%E2%89%A522.5-informational)
+[![XERJ Corpus Hub contributor](https://hub.xerj.org/contributors/surlymochan/badge.svg)](https://hub.xerj.org/contributors/surlymochan)
+
+**Claimed ≠ verified.**
+
+evalid is a runtime verification layer for coding agents. Your agent says the tests pass — evalid seals the run, replays it from a clean snapshot, and shows you the gap between what was claimed and what actually passed.
+
+- **Sealed evidence, replayed — not re-asked.** When tests run, the whole tree is hashed and sealed together with the command, exit code, and output. Verification copies the sealed snapshot to a clean directory and actually re-runs it, comparing exit codes and failure classes — and in the green phase, test files must hash-match the seal. If the agent edited the tests after the run, the seal breaks.
+- **A zero-judge pipeline.** Spec clarification, code review, and test verification are plain deterministic code — no LLM judge anywhere in the chain, zero tokens spent on checking, every stage leaving an artifact you can inspect. Ambiguous specs are blocked before the loop starts; changes without test follow-through are blocked at review; unproven claims are blocked at replay.
+- **The claim-vs-evidence ledger, with the cost attached.** Every delivery carries one record: what was claimed, what the replay showed, and what each phase cost in time and tokens — and when the phase ledger disagrees with wall-clock time by more than 5%, it flags itself as untrustworthy.
+
+> An LLM judge can be persuaded, is non-deterministic, and bills you for every opinion. A replay can't be talked out of anything: same sealed snapshot, same commands, same exit codes — or the claim doesn't ship.
+
+> **What the seal does and doesn't prove.** The seal is a content hash, not a cryptographic signature. A fully self-consistent forged snapshot could, in theory, still pass — we state this openly rather than oversell. What the seal does guarantee is narrower and real: the tests must actually have run at seal time, the test files at verification time are byte-identical to the ones that ran, and any tampering after the run breaks the hash and shows up. It is built to catch misreporting and test tampering — the failure modes agents actually exhibit — not to defeat a determined forger.
+
+Underneath, a workflow engine (spec → implement → verify loops with per-phase budgets) produces the work that gets verified. It's the chassis, not the headline: verification is the product. The `xflow` CLI runs the same workflows from Codex, Claude Code, Cursor, OpenCode, Gemini or a plain terminal.
+
+_Renamed from OpenFlow in v0.3.1 (2026-11-04); old repository and package links redirect._
+
+## Benchmarks: evalid vs superpowers (2026-10-17)
+
+_These results were measured before the rename, when the project was named OpenFlow; product names in this section have been unified to the current names (evalid, swrm), and all versions and figures are the original measurements._
+
+For the synthetic and evidence-check results below, the same model (`grok-4.7-high` via local `cursor-agent`) and same
+harness are used for every arm; the SWE-bench block uses DeepSeek via Pi. Pinned: evalid 0.2.1,
+[superpowers](https://github.com/obra/superpowers) v7.0.0 (`bb92a77`). Scripts, raw results and methodology:
+[`bench/`](bench/README.md) and [`bench/e2e/`](bench/e2e/README.md). Losses are shown as measured.
+
+**Real GitHub issues (SWE-bench Verified), 2026-10-18.** Six hard instances from sympy, pytest and pylint; success
+means the official FAIL_TO_PASS and PASS_TO_PASS tests all pass. Every arm runs on the Pi coding agent with
+DeepSeek-V4.1-Flash (`deepseek-flash`), n=1 per cell, 20-minute cap; details and commands are in
+[`bench/swebench/`](bench/swebench/README.md).
+
+| Arm | Resolved (official tests) | Median wall-clock (IQR) | Median $ per task (DeepSeek list price) | Total $ (all runs) | n |
+| --- | --- | --- | --- | --- | --- |
+| Plain agent (baseline) | 3/6 (50%) | 124 s (110–175) | $0.0234 | $0.16 | 6 |
+| superpowers v7.0.0 | 2/6 (33%) | 154 s (136–340) | $0.0285 | $0.31 | 6 |
+| evalid 0.2.1 | 4/6 (67%) | 230 s (200–266) | $0.0412 | $0.23 | 6 |
+| swrm (Pi executor) | 5/6 (83%) | 228 s (187–254) | $0.0377 | $0.21 | 6 |
+
+**Caveats:** n=1 on 6 instances, so one or two resolves are within noise. The two workflow tools took about 1.8x the
+baseline's median wall-clock and cost 76% (evalid) / 61% (swrm) more per task. The first published table undercounted
+pytest because of a shared temp-dir race in grading; it is now fixed and regraded
+([regrade notes](bench/swebench/README.md#regrade-2026-10-18)). On the MiniMax-M3.1-Flash-Preview subset, baseline
+resolved 3/4 and superpowers 2/4; evalid had no clean MiniMax run because the plan's usage cap was hit.
+
+Raw per-run records and logs: [`bench/swebench/results/`](bench/swebench/results/). Optimization backlog from these
+results: [#5](https://github.com/surlymochan/evalid/issues/5).
+
+**Synthetic end-to-end tasks.** 8 small real tasks (4 JS, 4 Python), hidden acceptance tests applied only after the agent exits, 20-minute cap per run.
+
+| Arm | Success (hidden tests) | Median wall time (IQR) | Median tokens: total / input+output | n |
+| --- | --- | --- | --- | --- |
+| Plain agent (baseline) | 24/24 (100%) | 55 s (46–132) | 105k / 28k | 24 |
+| superpowers v7.0.0 | 24/24 (100%) | 187 s (121–299) | 312k / 59k | 24 |
+| evalid 0.2.1 | 11/12 (92%) | 985 s (920–1195) | 5.29M / 233k ¹ | 12 ² |
+
+On tasks this small the plain agent already succeeds every time, and evalid's full ceremony is the slowest and most
+token-hungry arm. 3 of 12 evalid runs hit the 20-minute cap while still running the workflow; their code already passed.
+¹ Tokens counted only for runs that reported usage (not the 3 capped runs). Most of "total" is cache reads.
+² 3 repeats on 2 tasks, 1 on the other 6, to save quota. Matched repeat-1 view (n=8 each): baseline 8/8, 57 s, 104k; superpowers 8/8, 154 s, 289k; evalid 8/8, 1008 s, 5.32M.
+
+**Catching fake TDD / verification evidence.** 36 scripted episodes: 25 faulty (red proof recorded after the fix, forged
+proofs, skipped tests, weak or wrong assertions, contradictory specs, …) and 11 honest ones. Of these, 12 are a held-out set written after the fixes.
+
+| Checker | Faulty caught | Honest wrongly rejected | Cost per check |
+| --- | --- | --- | --- |
+| evalid 0.2.1 gates (deterministic code) | 22/25 (88%): original 16/17, held-out 6/8 | 0/11 | ~0.1–0.2 s, no model tokens |
+| superpowers v7.0.0 skills + LLM judge | 25/25 (100%) | 1/11 | one agent call per check |
+| LLM judge, no skills | 25/25 (100%) ³ | 0/11 | one agent call per check |
+| evalid 0.2.0 (before this benchmark, original 24 only) | 10/17 (59%) | 3/7 | – |
+
+An LLM reviewer that is explicitly asked to audit an episode, and is allowed to rerun the tests, caught more than evalid's static gates did.
+evalid's gates are deterministic and cost no model calls, but they still miss plausible-but-wrong expected values,
+assertions in callbacks that never run, and behavior changes hidden inside a declared refactor.
+³ 1 of the 11 honest runs timed out and counts as incorrect. LLM arms used 1 run per scenario.
+
+Prompt-based workflows *ask* an agent to write the failing test first, run the suite and keep notes. evalid
+*checks* that it did. Workflows are YAML phases made of small deterministic "atoms". Gates are code: a
+red/green TDD proof, a test-quality review, artifact checks, and archive ordering. The `xflow` CLI runs the
+same workflows from Codex, Claude Code, Cursor, OpenCode, Gemini or a plain terminal.
+
+![evalid demo: a red proof must fail, a green proof must pass, and a faked red proof is rejected](docs/assets/demo.gif)
+
+<sub>Recorded from a real run of the CLI in an empty project. The last two commands show an agent "recording" a
+red proof after the fix, and the gate rejecting it.</sub>
+
+## Why
+
+Agent workflow kits are good at shaping behaviour, but the checks only live in prose. Agents skip the red
+step, write tests that assert nothing, lose context between sessions, and open PRs before verification has
+finished. evalid adds a harder layer underneath:
+
+- **Workflow as data.** `workflows/yolo.yaml` and `workflows/corps.yaml` are validated against an atom registry before they run.
+- **TDD proof checked by code.**
+  - `I6a` snapshots the tree and binds the command, exit code, output hash, and failure class. `I6b` recomputes that binding and re-runs the command on the snapshot. A hand-edited proof does not pass unless that snapshot really exits as claimed.
+  - Red and green must target the same test, or green must be a superset of the red test. Skipped, todo, and focused tests do not count as red or green.
+  - `I6c.tdd.quality_review` rejects code changes that have:
+    - no meaningful test changes,
+    - empty assertions,
+    - snapshot-only tests,
+    - mock-heavy tests without real assertions,
+    - only a type or existence assertion (`typeof`, `!== undefined`).
+  - `tdd_mode` in `status.json` (or `--tdd-mode`) may be `red-green` (default), `refactor`, or `docs`. Refactor requires the existing tests to pass before and after, with no assertion removed. Docs allows a non-code diff and does not require a red proof. Declaring docs while changing code, or refactor while deleting assertions, fails.
+- **Context that survives sessions.**
+  - The `.xflow/GOAL.md` project anchor.
+  - `HANDOFF.md` and `AHA.md`.
+  - Per-change `status.json` and proof files under `specs/changes/<id>/`.
+- **Executable finish.** Verify → archive → commit/push → PR ordering is a tested atom, not a checklist.
+
+## Features
+
+- **YAML workflows, validated before they run** — phases are data (`workflows/*.yaml`), checked against an atom registry before execution.
+- **Red/green TDD proof gates enforced by code** — proofs bind the command, exit code and output hash, and are re-verified on a snapshot; hand-edited or faked proofs are rejected, and skipped, todo or focused tests do not count.
+- **Automatic test-quality review** — changes with no meaningful test changes, empty assertions, snapshot-only tests or mock-heavy tests without real assertions are rejected.
+- **Context that survives sessions** — `.xflow/GOAL.md`, `HANDOFF.md` and per-change `status.json` carry state across sessions and agents.
+- **Executable finish line** — verify → archive → commit/push → PR ordering is a tested workflow atom, not a prose checklist.
+- **One CLI for multiple agents** — the `xflow` CLI drives Codex, Claude Code and Cursor from the same workflow files.
+
+## 3 Minute Start
+
+Requires Node.js ≥ 22.5 (for `node:sqlite`) and Python 3 (atoms are small Python scripts).
+
+```bash
+npm install -g evalid     # installs the `xflow` CLI
+cd your-project
+xflow init --project-root .               # writes .as-xflow/config.json
+xflow goal set "Ship the next verified change" --project-root .
+xflow goal audit --project-root . --json
+xflow doctor --project-root .
+xflow workflow validate yolo --project-root .
+xflow guide                               # the delivery loop in one screen
+```
+
+For a source checkout:
+
+```bash
+git clone https://github.com/surlymochan/evalid.git
+cd evalid
+npm ci && npm test
+node bin/xflow.js quickstart
+node bin/xflow.js evaluate
+node bin/xflow.js compare codex-goal
+node bin/xflow.js launch copy
+```
+
+## Run a change
+
+```bash
+git checkout -b feat/example-change
+xflow workflow run yolo \
+  --title "Example change" --change-type backend \
+  --tdd-red-command  "npm test -- --grep new-behavior" \
+  --tdd-green-command "npm test"
+```
+
+The lite track runs `tdd-red -> execute -> tdd-green -> verify -> archive` and stops at human or LLM gates. Every
+phase leaves evidence in `specs/changes/<change-id>/`. Add `--dry-run` to print the phases, atoms and gates
+first.
+
+You can also run a single gate directly, as in the demo:
+
+```bash
+xflow atom run I6a.tdd.run --change-id my-change --phase red   --test-command "npm test"
+xflow atom run I6a.tdd.run --change-id my-change --phase green --test-command "npm test"
+xflow atom run I6b.tdd.proof_validate --change-id my-change --phase red
+```
+
+## Evaluate it
+
+```bash
+xflow quickstart        # copy-paste-safe first-run commands
+xflow evaluate          # one-shot brief: quality score, comparisons, blockers, next actions
+xflow assess            # scorecard for goal, yolo, corps and the skill family
+xflow score             # competitive readiness score with evidence per dimension
+xflow demo launch       # goal -> yolo and goal -> corps proof paths
+xflow launch dossier    # one-page narrative backed by the same audit
+xflow launch copy       # claim-safe announcement copy plus forbidden phrases
+xflow release status    # the single next release blocker
+```
+
+<details>
+<summary>Release-owner gates (fail until real adoption / npm evidence exists)</summary>
+
+```bash
+xflow adoption trial --name <team-or-project> --source <tracker-or-pr> --track yolo
+xflow adoption status --json
+xflow package preflight --check-registry --check-auth --json
+xflow package audit --check-registry --json
+xflow launch audit --pre-publish --strict --json
+```
+
+</details>
+
+## Choose a track
+
+| Track | Use when | Entry |
+| --- | --- | --- |
+| `goal` | You want a project-level direction anchor that later work reads and audits | `xflow goal set`, `xflow goal audit` |
+| `plan` | You only need a reusable implementation plan | `xflow:plan` skill |
+| `yolo` | Backend, docs, infra, small UI; low ceremony | `xflow workflow run yolo` |
+| `corps` | Product/UI-heavy or multi-agent work with governed proof | `xflow corps --explain`, then `xflow corps` and `xflow proof --track corps` |
+| `ralph` | Keep implementing, verifying and repairing until the completion evidence is clean | `xflow:ralph` skill |
+
+Not sure which one fits? Run `xflow intake recommend --change-id <id> --title "<title>" --description "<text>"`.
+
+## How it fits together
+
+```
+workflows/*.yaml ──validate──► atoms/registry.json ──► xflow/atoms/*.py   (deterministic steps)
+        │                                                    │
+        ▼                                                    ▼
+  xflow workflow run ──► gates: deterministic | artifact-verify | human | llm
+        │
+        ▼
+  specs/changes/<id>/   status.json · tdd/red-0.json · tdd/green-0.json · quality-0.json · logs/*.ndjson
+```
+
+- `bin/xflow.js`: the CLI. `src/`: workflow loader, runner, policy overlays, scoring.
+- `xflow/`: the portable skill family (`xflow:plan`, `xflow:yolo`, `xflow:corps`, `xflow:ralph`, …) for agent hosts.
+- `schemas/`: JSON schemas for workflows, proofs and policy overlays.
+
+## Compared with other tools
+
+evalid aims to complement prompt-level workflow kits and spec tools, not replace them. `xflow compare
+<superpowers|openspec|gstack|spec-kit|codex-goal>` prints a side-by-side with sources.
+
+<details>
+<summary>Full capability table</summary>
+
+| Surface | evalid Position |
+| --- | --- |
+| Goal alignment | Project-level `.xflow/GOAL.md` that yolo, corps, Ralph, handoff, and takein consume |
+| One-shot evaluation | `xflow evaluate` combines quality score, Codex goal / Superpowers / Super-Assistant comparison, ordinary and splash launch blockers, release-owner status, claim boundaries, next actions, and first-run path |
+| Quality assessment | `xflow assess` gives a public scorecard for goal, yolo, corps, and the skill family |
+| Launch demo | `xflow demo launch` shows goal -> yolo and goal -> corps proof paths |
+| Launch audit | `xflow launch audit` separates ready engine and adoption evidence from missing registry proof |
+| Launch dossier | `xflow launch dossier` prints the one-page public narrative backed by the same audit payload |
+| Launch claims | `xflow launch claims` separates what can be said publicly now from claims blocked by missing evidence |
+| Launch copy | `xflow launch copy` generates claim-safe announcement copy and forbidden phrases from the same evidence |
+| Package status | `xflow package status` gives a non-failing publish/install status summary |
+| Package audit | `xflow package audit` validates public npm registry evidence before splash-launch claims |
+| Release status | `xflow release status` gives the release owner one focused blocker, next action, package status, and handoff doc |
+| Adoption status | `xflow adoption status` gives a non-failing ordinary/splash adoption blocker summary |
+| Adoption trial | `xflow adoption trial` gives external teams the exact commands and evidence checklist before a trial packet is written |
+| Publish handoff | `docs/npm-publish-handoff.md` records the exact auth, preflight, publish, and post-publish gates |
+| TDD | Executable red/green proof plus quality review |
+| Workflow | YAML phases validated against atom registry |
+| Spec migration | OpenSpec mapping report plus deterministic `spec_delta_review.json` |
+| Local closure | Doctor, drift scan, release pack, archive ordering |
+| Team handoff | `HANDOFF.md`, `AHA.md`, reviewer guide |
+| Team adoption | Role model, CI boundary, PR acceptance, release ownership |
+| Cross-tool use | Plain CLI and skill docs for Codex, Claude Code, Cursor, OpenCode, Gemini |
+| Competitive proof | `xflow compare codex-goal`, `xflow compare superpowers`, `xflow compare super-assistant`, `xflow compare openspec`, `xflow compare gstack`, `xflow compare spec-kit`, benchmark docs |
+| Adoption proof | Minimal clean project runs in CI, plus public adoption-record templates |
+
+</details>
+
+## Docs
+
+<details>
+<summary>All guides</summary>
+
+- [Quickstart](docs/quickstart.md)
+- [Install and upgrade](docs/install-upgrade.md)
+- [Tooling matrix](docs/tooling-matrix.md)
+- [Compatibility notes](docs/compatibility.md)
+- [Demo proof](docs/demo-proof.md)
+- [Examples gallery](docs/examples-gallery.md)
+- [Public benchmark](docs/public-benchmark.md)
+- [Quality assessment](docs/quality-assessment.md)
+- [Launch demo](docs/launch-demo.md)
+- [Launch dossier](docs/launch-dossier.md)
+- [Corps operator guide](docs/corps-operator-guide.md)
+- [Superpowers comparison](docs/superpowers-comparison.md)
+- [Super-Assistant comparison](docs/super-assistant-comparison.md)
+- [Adoption evidence template](docs/adoption/README.md)
+- [Maintainer adoption record](docs/adoption/openflow-release-hardening.md)
+- [xflow Goal vs Codex native goal](docs/goal-vs-codex.md)
+- [Walkthrough](docs/walkthrough.md)
+- [OpenSpec migration](docs/openspec-migration.md)
+- [spec-kit benchmark](docs/spec-kit-benchmark.md)
+- [Integration matrix](docs/integrations.md)
+- [Public release gate](docs/public-release.md)
+- [npm publish handoff](docs/npm-publish-handoff.md)
+- [Release notes](RELEASE_NOTES.md)
+- [Adapter fixture](docs/fixtures/tracker-item.json)
+- [Methodology](docs/methodology.md)
+- [Reviewer guide](docs/reviewer-guide.md)
+- [Team adoption operating model](docs/team-adoption.md)
+- [Competitive benchmark](docs/competitive-benchmark.md)
+- [2026-05-25 benchmark report](docs/benchmark-evalid-vs-gstack-superpowers-openspec-2026-05-25.md)
+
+</details>
+
+## Development
+
+```bash
+npm ci
+npx playwright install chromium   # browser-evidence tests
+npm test                          # 270+ tests
+npm run drift:scan                # docs/skill/workflow drift checks
+npm run pack:check                # what would be published
+```
+
+Contributions are welcome: open an issue describing the workflow problem first, then a PR with tests. Keep
+claims in docs tied to a command that reproduces them.
+
+## License
+
+[MIT](LICENSE)

package/RELEASE_NOTES.md

@@ -0,0 +1,124 @@
+# Release Notes
+
+## 0.4.1 - Failed runs report phase costs; Node 24 test counts fail closed
+
+Patch release. Two fixes, both about the gate trusting evidence it had not actually read.
+
+**Failed runs now report where the time and tokens went (#22).** The per-phase cost report landed in 0.3.0 but only rode on successful runs. It now also rides on the `workflow_failed` execution-log event — a run that stops mid-way still reports its per-phase timing/token breakdown — and is persisted as a standalone artifact at `.as-xflow/phase-cost.json` on both completion and failure, so the breakdown is readable without mining the ndjson execution log. The accounting structure (phase sum vs wall clock within 5%) is covered by `test/phase-cost.test.js`. This lands item P1-1 from issue #5.
+
+**TDD gate reads test counts correctly on Node 24, and fails closed when it cannot (#23).** The gate parsed skip/todo counts only from the TAP reporter's summary lines. Node 24's built-in runner defaults to the spec reporter when piped, whose summary lines carry a different marker — an all-skipped/todo run on Node 24 exited 0, parsed as zero skips, and was waved through as green. The parser now reads the runner's summary block in both reporter formats (TAP and spec), requires the full node signature before treating a block as a node summary, and anchors summary lines at line start so test names or program output cannot forge counts. When a detected node summary cannot be fully parsed, the run classifies as `counts_unknown` and proof validation rejects it (`tests_counts_unknown`) instead of reading unknown counts as zero. Measured against pinned captured outputs: the new count-parsing tests pass 9/9 under Node 22 and 9/9 under Node 24; the end-to-end TDD proof suite passes 18/18 under both (it was 17/18 on Node 24 before this fix).
+
+With P1-1 landed here, the P0/P1 items tracked on the optimization backlog in #5 stand checked off as verified; the remaining longer-horizon items stay tracked there.
+
+## 0.4.0 - CR gate wired in, overhead cut 1, predictions direct
+
+**The CR gate now runs inside the pipeline (F1, #15).** In 0.3.0 `J6.cr.review` was registered and unit-tested but no workflow invoked it — the release notes said so, and this release closes that gap. The yolo `verify-consistency` phase and the corps `qa` phase run J6 on every change, after the spec-delta review and before the stage flips to verify, and the verdict record `cr/review-0.json` is a required artifact: a failing verdict stops the workflow instead of waving the change through to archive. The wiring is proven end-to-end, not just by unit tests: `test/cr-review-wiring.test.js` drives the real yolo verify phase through `WorkflowExecutor` against a git fixture — an aligned, tested change passes with a verdict record on disk, and a code change with no test follow is stopped at the phase with the `cr_code_without_test` finding.
+
+**Overhead cut 1 (F2, #17).** The lite (yolo) track seals the TDD red proof immediately after `plan` — the issue/branch ceremony no longer sits between planning and red-proof, and the planning pass authors the failing test, so plan and red share one agent pass. `xflow workflow run --title "..."` derives the change id from the title when the branch cannot provide one. TDD snapshots are scoped: inside a git work tree, I6a seals only the files that differ from HEAD plus the files the test command names, and I6b materializes the rest from the recorded HEAD at replay. Offline measurements on the SWE-bench sympy tree: the sealed manifest drops from 1,543 files / 164 KB to the changed files only (2 files / 197 B for a one-file change), and the snapshot copy from 26.9 MB to ~51 KB. Proofs sealed before this change keep verifying.
+
+Acceptance debt, stated plainly: the v0.4 acceptance line for F2 was a live rerun of the same 6 SWE-bench instances — wall-clock ≤ 1.3× the baseline median, total tokens ≤ 2.0M, resolved ≥ 4/6. The bench generation environment is not currently restored, so those three numbers are still owed; they are not claimed here. If the rerun misses the line when it lands, the remaining overhead work continues as v0.4.1 rather than being declared done.
+
+**Predictions come straight out of `run.py` (F3, #16).** The official SWE-bench predictions format (`instance_id`, `model_name_or_path`, `model_patch`) is now produced by the bench harness itself, with the P0-2 evidence filter re-applied on both the live and replay paths — the one-off converter outside the repo is retired. Regression evidence: all four pilot arms rebuilt through `run.py --export-predictions` are byte-identical to the hand-converted pilot files (`cmp` = 0, 24 instances).
+
+**`xflow init` writes `.gitignore` (F4, #18).** Init now creates or merges the project `.gitignore` for evalid local state (`.as-xflow/`, `specs/changes/`), appending only missing entries and never modifying existing lines — the first line of defense ahead of the diff filtering added in #8. This closes the second v0.3.0 known gap; both are now done.
+
+## 0.3.1 - Renamed: OpenFlow → evalid
+
+Same code as 0.3.0, new name and positioning. The project is now **evalid** — a runtime verification layer for coding agents: it seals each test run and replays it from a clean snapshot, so a claim only ships if the evidence replays green. Repository: `surlymochan/evalid` (old `openflow` links redirect); npm package: `evalid`. The `xflow` CLI name is unchanged in this release.
+
+## 0.3.0 - Three-stage gates: SPEC, CR, Test
+
+OpenFlow's gates now cover the three stages where agent work usually goes wrong, each with deterministic checks instead of another model's opinion.
+
+**SPEC clarification gate (`J4d.spec_clarify.gate`, #11).** Ambiguity is cheapest before the loop starts. J4d blocks specs whose MUST clauses carry no quantified anchor (a clause whose value is itself a vague word does not count as anchored), scenarios without WHEN/THEN, vague wording in English or Chinese, and TBD/placeholder text. Every finding comes with a concrete question a human can answer. False-positive control is deliberate: MUST NOT clauses are exempt, already-quantified lines are not flagged, and code blocks and headings are not scanned. Ships as `xflow spec clarify` and in the yolo/corps proposal-consistency stage.
+
+**Code review gate (`J6.cr.review`, #10).** A diff fails review when nothing in it aligns with the spec, when behaviour changes ship without tests (an explicit `cr_allow_untested` status marker is the only, auditable, exemption), when assertions decrease net, or when test files were edited to make a red run look green. A loop budget (default 10 files / 400 changed lines) warns over budget and fails at 2x. Known gap: J6 is registered in the atom registry (now 85 atoms) and covered by its own tests, but is not yet wired into the yolo/corps verify stage — that wiring is planned for v0.4.
+
+**Test gate v2 (`I6c`, #12) and per-phase cost accounting.** I6c now also rejects tests whose expected value is computed from the implementation under test, assertions inside callbacks that never fire, and self-comparisons that are always true — each check ships with a passing counter-example so honest tests are not blocked. `src/core/phase-cost.js` records wall-clock and tokens per phase; a run whose phase sum diverges from total wall-clock by more than 5% is flagged `within_tolerance: false`, so cost claims stay auditable.
+
+**Clean benchmark patches (#8).** The SWE-bench harness generated `solution.diff` with the whole `specs/` evidence tree inside: one real openflow-arm patch was 17.0 MB, of which 99.9% was evidence files. Generation now excludes `specs/`, a harness-side filter strips it as a second line of defence, and regrade applies the same filter to old patches. The same patch is now 24 KB. `UV_CACHE_DIR` is also per-run now, and the repo gained `CONTRIBUTING.md` plus issue/PR templates.
+
+**CI fix (#9).** The verify job died at "Install Playwright" because `ubuntu-latest` had rolled to 26.04, which Playwright 1.59.1 does not support. The runner is pinned to `ubuntu-24.04`; upgrading Playwright itself is a separate, larger change.
+
+Also known: `xflow init` does not yet write OpenFlow entries into the user project's `.gitignore` (planned for v0.4).
+
+## 0.2.2 - Benchmark docs
+
+No CLI or gate changes: this release only updates documentation. The README now leads with the SWE-bench Verified result on DeepSeek-V4.1-Flash: OpenFlow resolved 4/6 and the plain baseline resolved 3/6 (n=1 per cell). It also states the caveats: n=1 on 6 instances puts one or two resolves within noise, and the two workflow tools took about 1.8x the baseline's median wall-clock and cost 76% (OpenFlow) / 61% (openswarm) more per task. The first published table undercounted pytest because of a shared temp-dir race in grading; it is now fixed and regraded, with notes in [bench/swebench/README.md](bench/swebench/README.md#regrade-2026-10-18). The optimization backlog from these results is tracked in [issue #5](https://github.com/surlymochan/openflow/issues/5).
+
+## 0.2.1 - Tighter TDD and spec gates
+
+Proof files are no longer trusted as JSON alone. `I6a` records a snapshot and a digest over the command, exit code, output hash, failure class, and file hashes. `I6b` recomputes that digest and re-runs the command on the snapshot. An agent who can rewrite the tree can still produce a self-consistent failing snapshot; a hand edit of the proof cannot.
+
+`I6c` requires red and green to target the same test (or a green superset), rejects skip/todo/only on the changed tests, and rejects a type-only assertion when it is the only assertion. A plausible but wrong expected value is still not decidable statically. `J4b` rejects contradictory `MUST`/`SHALL` values for the same subject.
+
+`tdd_mode` may be `red-green` (default), `refactor`, or `docs`, via `status.json` or `--tdd-mode`. Refactor checks that existing tests pass before and after and that assertions were not removed. Docs checks that the diff stays off code and tests.
+
+## 0.2.0 - Relaunch (first npm release)
+
+OpenFlow is published to npm as **`@surlymochan/openflow`**. The CLI is still `xflow`. The unscoped
+`openflow` name on npm belongs to an unrelated project.
+
+### Highlights
+
+- **Policy overlays.** `xflow policy validate` checks personal and team policy layered over the built-in
+  workflows. Policy, notify, storage, plugin and experience-digest atoms ship with it.
+- **Adaptive intake.** `xflow intake recommend` reads a demand profile (title, description, repo count) and
+  recommends `yolo` or `corps` plus gate overlays. It writes `workflow_recommendation.json`.
+- **Execution profiles.** `xflow profile show|set` handles personal and team execution modes and provider
+  preferences.
+- **Comparisons.** `xflow compare super-assistant` is new, next to the Superpowers, OpenSpec, gstack,
+  spec-kit and Codex goal comparisons.
+- **New README.** It covers the problem, a demo GIF recorded from a real run, a 3-minute start and
+  single-gate usage (`xflow atom run I6a.tdd.run ...`).
+
+### Packaging
+
+- `package.json` now has `repository`, `homepage`, `bugs`, `keywords`, `author`,
+  `engines.node >= 22.5` (`node:sqlite`) and `publishConfig.access = public`.
+- Install docs and the CLI's copy-paste commands now use `npm install -g @surlymochan/openflow`.
+- `xflow serve` no longer points to an unpublished server repository.
+- Fixed an uncaught `EPIPE` in `agent_invoke` when an agent CLI exits without reading stdin.
+
+### Verified for this release
+
+- `npm test`: 275 passed, 1 skipped, 0 failed (Node 22, Playwright Chromium installed).
+- `npm run drift:scan`: 9/9.
+- `npm publish --dry-run`: `@surlymochan/openflow@0.2.0`, 187 files, about 321 kB.
+- Clean install from the packed tarball into an empty project: `xflow init`, `goal set`, `doctor` and
+  `workflow validate yolo` all pass.
+
+## 0.1.0 - Public Readiness Candidate
+
+This release turns `openflow` from a local skill/workflow repo into a package-ready workflow runtime for AI coding agents.
+
+### Highlights
+
+- CLI package entry: `xflow` is exposed through `bin/xflow.js`.
+- First-run loop: `xflow guide`, `xflow demo clean`, `xflow init`, `xflow doctor`, and `xflow workflow validate yolo`.
+- Executable workflow data: `workflows/yolo.yaml` and `workflows/corps.yaml` are schema-validated.
+- TDD enforcement: red proof, green proof, and changed-test quality review are separate gates.
+- Spec durability: `xflow spec delta` and `xflow spec openspec-map` create local review/migration artifacts.
+- Competitive proof: `xflow score`, `xflow compare openspec`, and `xflow compare spec-kit`.
+- Adapter entrypoints: `xflow adapter import-file` and `xflow adapter github-issue` import external tracker context into local change artifacts.
+- Public release gate: `npm run publish:check` runs tests, drift scan, skill diff, package dry-run, and npm publish dry-run.
+
+### Adapter Commands
+
+```bash
+xflow adapter import-file --input docs/fixtures/tracker-item.json --json
+xflow adapter github-issue --repo owner/name --issue 123 --project-root .
+```
+
+Adapters are intentionally thin. They copy source context into `specs/changes/<change-id>/proposal.md` and `status.json`; they do not make GitHub, Linear, Jira, or any tracker the workflow source of truth.
+
+### Required Proof Before Real Publish
+
+```bash
+npm run publish:check
+xflow score
+xflow compare openspec
+xflow compare spec-kit
+```
+
+`npm publish --access public` is the only remaining registry-mutating step. It requires an authenticated npm session and should only run after the dry-run gate is green.